2024-02-06T21:21:23.954 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-06T21:21:23.954 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974 --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} --name lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi --verbose --owner scheduled_lflores@teuthology -- /home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974/orig.config.yaml 2024-02-06T21:21:23.994 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-06T21:21:24.073 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974 branch: wip-yuri4-testing-2024-02-05-0849 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7548974' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri4-testing-2024-02-05-0849 ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 633ab857b9926af935a3e6291c3e1d9251aca357 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 633ab857b9926af935a3e6291c3e1d9251aca357 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2024-02-05-0849-no-mon-cluster-logging sha1: 2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc owner: scheduled_lflores@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 4200 sha1: 633ab857b9926af935a3e6291c3e1d9251aca357 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2024-02-05-0849-no-mon-cluster-logging suite_path: /home/teuthworker/src/github.com_ljflores_ceph_2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc/qa suite_relpath: qa suite_repo: https://github.com/ljflores/ceph.git suite_sha1: 2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc targets: smithi120.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBERq3tD9axCtMLTUR91xYMSSyspJePB3jBEHX7Vx31cW7gZaKDEvhcIP2WBNlHrWqXXnbvMeAgpeFcKGIFWh2wY= smithi135.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE6IG63DZKKtm3VBcdAQFlU7KcQl66IeJPSg7EJNL9LVQtVfynSBOJoPR4umbHvTxWK70Dd61OqnVyNHcVGH45g= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-06_20:55:47 tube: smithi user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-06T21:21:24.073 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ljflores_ceph_2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc/qa; will attempt to use it 2024-02-06T21:21:24.074 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ljflores_ceph_2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc/qa/tasks 2024-02-06T21:21:24.074 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-06T21:21:24.075 INFO:teuthology.task.internal:Checking packages... 2024-02-06T21:21:24.099 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '633ab857b9926af935a3e6291c3e1d9251aca357' 2024-02-06T21:21:24.099 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-06T21:21:24.099 INFO:teuthology.packaging:ref: None 2024-02-06T21:21:24.100 INFO:teuthology.packaging:tag: None 2024-02-06T21:21:24.100 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-02-05-0849 2024-02-06T21:21:24.100 INFO:teuthology.packaging:sha1: 633ab857b9926af935a3e6291c3e1d9251aca357 2024-02-06T21:21:24.100 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri4-testing-2024-02-05-0849 2024-02-06T21:21:24.313 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1269.g633ab857 2024-02-06T21:21:24.315 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-06T21:21:24.321 INFO:teuthology.task.internal:no buildpackages task found 2024-02-06T21:21:24.321 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-06T21:21:24.326 INFO:teuthology.task.internal:Saving configuration 2024-02-06T21:21:24.341 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-06T21:21:24.346 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-06T21:21:24.375 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-06 21:16:16.060020', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBERq3tD9axCtMLTUR91xYMSSyspJePB3jBEHX7Vx31cW7gZaKDEvhcIP2WBNlHrWqXXnbvMeAgpeFcKGIFWh2wY='} 2024-02-06T21:21:24.399 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-06 21:16:16.062115', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE6IG63DZKKtm3VBcdAQFlU7KcQl66IeJPSg7EJNL9LVQtVfynSBOJoPR4umbHvTxWK70Dd61OqnVyNHcVGH45g='} 2024-02-06T21:21:24.399 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-06T21:21:24.404 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-02-06T21:21:24.404 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-02-06T21:21:24.404 INFO:teuthology.run_tasks:Running task console_log... 2024-02-06T21:21:24.472 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa01cd52f70>, signals=[15]) 2024-02-06T21:21:24.472 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-06T21:21:24.478 INFO:teuthology.task.internal:Opening connections... 2024-02-06T21:21:24.478 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi120.front.sepia.ceph.com 2024-02-06T21:21:24.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T21:21:24.570 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi135.front.sepia.ceph.com 2024-02-06T21:21:24.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T21:21:24.657 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-06T21:21:24.663 DEBUG:teuthology.orchestra.run.smithi120:> uname -m 2024-02-06T21:21:24.680 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2024-02-06T21:21:24.680 DEBUG:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2024-02-06T21:21:24.737 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="CentOS Stream" 2024-02-06T21:21:24.737 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="9" 2024-02-06T21:21:24.737 INFO:teuthology.orchestra.run.smithi120.stdout:ID="centos" 2024-02-06T21:21:24.737 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="rhel fedora" 2024-02-06T21:21:24.737 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="9" 2024-02-06T21:21:24.737 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el9" 2024-02-06T21:21:24.737 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-06T21:21:24.737 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31" 2024-02-06T21:21:24.737 INFO:teuthology.orchestra.run.smithi120.stdout:LOGO="fedora-logo-icon" 2024-02-06T21:21:24.737 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-06T21:21:24.737 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://centos.org/" 2024-02-06T21:21:24.738 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-06T21:21:24.738 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-06T21:21:24.738 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-06T21:21:24.738 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2024-02-06T21:21:24.765 DEBUG:teuthology.orchestra.run.smithi135:> uname -m 2024-02-06T21:21:24.782 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2024-02-06T21:21:24.782 DEBUG:teuthology.orchestra.run.smithi135:> cat /etc/os-release 2024-02-06T21:21:24.837 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="CentOS Stream" 2024-02-06T21:21:24.837 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="9" 2024-02-06T21:21:24.838 INFO:teuthology.orchestra.run.smithi135.stdout:ID="centos" 2024-02-06T21:21:24.838 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE="rhel fedora" 2024-02-06T21:21:24.838 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="9" 2024-02-06T21:21:24.838 INFO:teuthology.orchestra.run.smithi135.stdout:PLATFORM_ID="platform:el9" 2024-02-06T21:21:24.838 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-06T21:21:24.838 INFO:teuthology.orchestra.run.smithi135.stdout:ANSI_COLOR="0;31" 2024-02-06T21:21:24.838 INFO:teuthology.orchestra.run.smithi135.stdout:LOGO="fedora-logo-icon" 2024-02-06T21:21:24.838 INFO:teuthology.orchestra.run.smithi135.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-06T21:21:24.838 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="https://centos.org/" 2024-02-06T21:21:24.838 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-06T21:21:24.838 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-06T21:21:24.838 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-06T21:21:24.839 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2024-02-06T21:21:24.866 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-06T21:21:24.874 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-06T21:21:24.899 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-06T21:21:24.900 DEBUG:teuthology.orchestra.run.smithi120:> test '!' -e /home/ubuntu/cephtest 2024-02-06T21:21:24.902 DEBUG:teuthology.orchestra.run.smithi135:> test '!' -e /home/ubuntu/cephtest 2024-02-06T21:21:24.917 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-06T21:21:24.922 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-06T21:21:24.922 DEBUG:teuthology.orchestra.run.smithi120:> test -z $(ls -A /var/lib/ceph) 2024-02-06T21:21:24.958 DEBUG:teuthology.orchestra.run.smithi135:> test -z $(ls -A /var/lib/ceph) 2024-02-06T21:21:25.073 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-06T21:21:25.123 INFO:teuthology.run_tasks:Running task kernel... 2024-02-06T21:21:25.138 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-06T21:21:25.138 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-06T21:21:25.138 DEBUG:teuthology.orchestra.run.smithi120:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-06T21:21:25.139 DEBUG:teuthology.orchestra.run.smithi135:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-06T21:21:25.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-06T21:21:25.154 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2024-02-06T21:21:25.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-06T21:21:25.155 DEBUG:teuthology.orchestra.run.smithi120:> uname -r 2024-02-06T21:21:25.209 INFO:teuthology.orchestra.run.smithi135.stdout:5.14.0-361.el9.x86_64 2024-02-06T21:21:25.209 INFO:teuthology.task.kernel:Running kernel on smithi135: 5.14.0-361.el9.x86_64 2024-02-06T21:21:25.209 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum install -y kernel 2024-02-06T21:21:25.212 INFO:teuthology.orchestra.run.smithi120.stdout:5.14.0-361.el9.x86_64 2024-02-06T21:21:25.212 INFO:teuthology.task.kernel:Running kernel on smithi120: 5.14.0-361.el9.x86_64 2024-02-06T21:21:25.213 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum install -y kernel 2024-02-06T21:21:27.340 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - AppStream 14 MB/s | 6.6 MB 00:00 2024-02-06T21:21:27.405 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS-9 - AppStream 12 MB/s | 6.6 MB 00:00 2024-02-06T21:21:28.932 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-02-06T21:21:29.005 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-02-06T21:21:30.339 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - BaseOS 7.4 MB/s | 8.0 MB 00:01 2024-02-06T21:21:30.404 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - BaseOS 7.4 MB/s | 8.0 MB 00:01 2024-02-06T21:21:33.317 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-02-06T21:21:34.282 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - AppStream 6.5 MB/s | 19 MB 00:02 2024-02-06T21:21:38.460 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 5.9 MB 00:01 2024-02-06T21:21:39.866 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 15 kB 00:00 2024-02-06T21:21:46.932 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - CRB 663 kB/s | 5.9 MB 00:09 2024-02-06T21:21:48.357 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 15 kB 00:00 2024-02-06T21:21:59.477 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 20 MB 00:19 2024-02-06T21:22:04.574 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 5.2 kB/s | 1.7 kB 00:00 2024-02-06T21:22:06.890 INFO:teuthology.orchestra.run.smithi135.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-06T21:22:06.890 INFO:teuthology.orchestra.run.smithi135.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-06T21:22:06.992 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:22:07.000 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:22:07.000 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-02-06T21:22:07.000 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:22:07.000 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:22:07.000 INFO:teuthology.orchestra.run.smithi135.stdout: kernel x86_64 5.14.0-417.el9 CentOS-BaseOS 6.0 M 2024-02-06T21:22:07.000 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-02-06T21:22:07.001 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-core x86_64 5.14.0-417.el9 CentOS-BaseOS 21 M 2024-02-06T21:22:07.001 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules x86_64 5.14.0-417.el9 CentOS-BaseOS 39 M 2024-02-06T21:22:07.001 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules-core x86_64 5.14.0-417.el9 CentOS-BaseOS 33 M 2024-02-06T21:22:07.001 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:22:07.001 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:22:07.001 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:22:07.001 INFO:teuthology.orchestra.run.smithi135.stdout:Install 4 Packages 2024-02-06T21:22:07.001 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:22:07.002 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 99 M 2024-02-06T21:22:07.002 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 122 M 2024-02-06T21:22:07.002 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:22:07.559 INFO:teuthology.orchestra.run.smithi135.stdout:(1/4): kernel-5.14.0-417.el9.x86_64.rpm 11 MB/s | 6.0 MB 00:00 2024-02-06T21:22:08.468 INFO:teuthology.orchestra.run.smithi135.stdout:(2/4): kernel-core-5.14.0-417.el9.x86_64.rpm 14 MB/s | 21 MB 00:01 2024-02-06T21:22:08.476 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 20 MB 00:19 2024-02-06T21:22:09.576 INFO:teuthology.orchestra.run.smithi135.stdout:(3/4): kernel-modules-5.14.0-417.el9.x86_64.rpm 15 MB/s | 39 MB 00:02 2024-02-06T21:22:10.035 INFO:teuthology.orchestra.run.smithi135.stdout:(4/4): kernel-modules-core-5.14.0-417.el9.x86_6 14 MB/s | 33 MB 00:02 2024-02-06T21:22:10.038 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:22:10.038 INFO:teuthology.orchestra.run.smithi135.stdout:Total 33 MB/s | 99 MB 00:03 2024-02-06T21:22:10.038 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:22:10.426 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:22:10.426 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:22:11.443 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:22:11.443 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:22:13.424 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 10 kB/s | 1.7 kB 00:00 2024-02-06T21:22:13.817 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:22:14.534 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-modules-core-5.14.0-417.el9.x86_64 1/4 2024-02-06T21:22:14.665 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-core-5.14.0-417.el9.x86_64 2/4 2024-02-06T21:22:15.722 INFO:teuthology.orchestra.run.smithi120.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-06T21:22:15.722 INFO:teuthology.orchestra.run.smithi120.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-06T21:22:15.815 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:22:15.825 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:22:15.826 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-02-06T21:22:15.826 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:22:15.826 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:22:15.826 INFO:teuthology.orchestra.run.smithi120.stdout: kernel x86_64 5.14.0-417.el9 CentOS-BaseOS 6.0 M 2024-02-06T21:22:15.826 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-02-06T21:22:15.826 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-core x86_64 5.14.0-417.el9 CentOS-BaseOS 21 M 2024-02-06T21:22:15.826 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-modules x86_64 5.14.0-417.el9 CentOS-BaseOS 39 M 2024-02-06T21:22:15.826 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-modules-core x86_64 5.14.0-417.el9 CentOS-BaseOS 33 M 2024-02-06T21:22:15.826 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:22:15.826 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:22:15.826 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:22:15.826 INFO:teuthology.orchestra.run.smithi120.stdout:Install 4 Packages 2024-02-06T21:22:15.826 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:22:15.826 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 99 M 2024-02-06T21:22:15.826 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 122 M 2024-02-06T21:22:15.827 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:22:16.191 INFO:teuthology.orchestra.run.smithi120.stdout:(1/4): kernel-5.14.0-417.el9.x86_64.rpm 17 MB/s | 6.0 MB 00:00 2024-02-06T21:22:16.775 INFO:teuthology.orchestra.run.smithi120.stdout:(2/4): kernel-core-5.14.0-417.el9.x86_64.rpm 22 MB/s | 21 MB 00:00 2024-02-06T21:22:17.457 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-core-5.14.0-417.el9.x86_64 2/4 2024-02-06T21:22:17.650 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-modules-5.14.0-417.el9.x86_64 3/4 2024-02-06T21:22:17.658 INFO:teuthology.orchestra.run.smithi120.stdout:(3/4): kernel-modules-core-5.14.0-417.el9.x86_6 23 MB/s | 33 MB 00:01 2024-02-06T21:22:18.125 INFO:teuthology.orchestra.run.smithi120.stdout:(4/4): kernel-modules-5.14.0-417.el9.x86_64.rpm 17 MB/s | 39 MB 00:02 2024-02-06T21:22:18.128 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:22:18.129 INFO:teuthology.orchestra.run.smithi120.stdout:Total 43 MB/s | 99 MB 00:02 2024-02-06T21:22:18.129 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:22:18.513 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:22:18.513 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:22:19.531 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:22:19.532 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:22:21.950 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:22:22.363 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-modules-5.14.0-417.el9.x86_64 3/4 2024-02-06T21:22:22.472 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-5.14.0-417.el9.x86_64 4/4 2024-02-06T21:22:22.644 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : kernel-modules-core-5.14.0-417.el9.x86_64 1/4 2024-02-06T21:22:22.786 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : kernel-core-5.14.0-417.el9.x86_64 2/4 2024-02-06T21:22:25.531 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-core-5.14.0-417.el9.x86_64 2/4 2024-02-06T21:22:25.843 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : kernel-modules-5.14.0-417.el9.x86_64 3/4 2024-02-06T21:22:27.203 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-modules-core-5.14.0-417.el9.x86_64 4/4 2024-02-06T21:22:30.498 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-modules-5.14.0-417.el9.x86_64 3/4 2024-02-06T21:22:30.697 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : kernel-5.14.0-417.el9.x86_64 4/4 2024-02-06T21:22:35.408 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-modules-core-5.14.0-417.el9.x86_64 4/4 2024-02-06T21:22:56.722 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-core-5.14.0-417.el9.x86_64 4/4 2024-02-06T21:22:56.722 INFO:teuthology.orchestra.run.smithi135.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-06T21:22:56.722 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:22:56.793 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-modules-5.14.0-417.el9.x86_64 4/4 2024-02-06T21:22:57.223 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-5.14.0-417.el9.x86_64 4/4 2024-02-06T21:22:57.223 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-5.14.0-417.el9.x86_64 1/4 2024-02-06T21:22:57.223 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-core-5.14.0-417.el9.x86_64 2/4 2024-02-06T21:22:57.223 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-modules-5.14.0-417.el9.x86_64 3/4 2024-02-06T21:22:57.666 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-modules-core-5.14.0-417.el9.x86_64 4/4 2024-02-06T21:22:57.667 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:22:57.667 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:22:57.667 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-5.14.0-417.el9.x86_64 2024-02-06T21:22:57.667 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-core-5.14.0-417.el9.x86_64 2024-02-06T21:22:57.667 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules-5.14.0-417.el9.x86_64 2024-02-06T21:22:57.667 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules-core-5.14.0-417.el9.x86_64 2024-02-06T21:22:57.667 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:22:57.667 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:22:58.224 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q kernel | sort -rV | head -n 1 2024-02-06T21:22:58.357 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-5.14.0-417.el9.x86_64 2024-02-06T21:22:58.358 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-417.el9.x86_64 2024-02-06T21:22:58.358 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-417.el9.x86_64 2024-02-06T21:22:58.358 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-06T21:22:58.358 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-06T21:22:58.358 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-06T21:22:58.358 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi135.front.sepia.ceph.com, path=None, version=distro) 2024-02-06T21:22:58.358 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q kernel | sort -rV | head -n 1 2024-02-06T21:22:58.467 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-5.14.0-417.el9.x86_64 2024-02-06T21:22:58.467 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-417.el9.x86_64 2024-02-06T21:22:58.467 DEBUG:teuthology.orchestra.run.smithi135:> sudo rpm -qi grub2-tools 2024-02-06T21:22:58.502 INFO:teuthology.orchestra.run.smithi135.stdout:Name : grub2-tools 2024-02-06T21:22:58.502 INFO:teuthology.orchestra.run.smithi135.stdout:Epoch : 1 2024-02-06T21:22:58.502 INFO:teuthology.orchestra.run.smithi135.stdout:Version : 2.06 2024-02-06T21:22:58.502 INFO:teuthology.orchestra.run.smithi135.stdout:Release : 68.el9 2024-02-06T21:22:58.502 INFO:teuthology.orchestra.run.smithi135.stdout:Architecture: x86_64 2024-02-06T21:22:58.502 INFO:teuthology.orchestra.run.smithi135.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-06T21:22:58.502 INFO:teuthology.orchestra.run.smithi135.stdout:Group : Unspecified 2024-02-06T21:22:58.502 INFO:teuthology.orchestra.run.smithi135.stdout:Size : 8264254 2024-02-06T21:22:58.502 INFO:teuthology.orchestra.run.smithi135.stdout:License : GPLv3+ 2024-02-06T21:22:58.502 INFO:teuthology.orchestra.run.smithi135.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-06T21:22:58.503 INFO:teuthology.orchestra.run.smithi135.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-06T21:22:58.503 INFO:teuthology.orchestra.run.smithi135.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-06T21:22:58.503 INFO:teuthology.orchestra.run.smithi135.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-06T21:22:58.503 INFO:teuthology.orchestra.run.smithi135.stdout:Packager : builder@centos.org 2024-02-06T21:22:58.503 INFO:teuthology.orchestra.run.smithi135.stdout:Vendor : CentOS 2024-02-06T21:22:58.503 INFO:teuthology.orchestra.run.smithi135.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-06T21:22:58.503 INFO:teuthology.orchestra.run.smithi135.stdout:Summary : Support tools for GRUB. 2024-02-06T21:22:58.503 INFO:teuthology.orchestra.run.smithi135.stdout:Description : 2024-02-06T21:22:58.503 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:22:58.503 INFO:teuthology.orchestra.run.smithi135.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-06T21:22:58.503 INFO:teuthology.orchestra.run.smithi135.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-06T21:22:58.503 INFO:teuthology.orchestra.run.smithi135.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-06T21:22:58.503 INFO:teuthology.orchestra.run.smithi135.stdout:hardware devices. 2024-02-06T21:22:58.503 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:22:58.504 INFO:teuthology.orchestra.run.smithi135.stdout:This subpackage provides tools for support of all platforms. 2024-02-06T21:22:58.504 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-06T21:22:58.504 INFO:teuthology.task.kernel:Updating grub on smithi135 to boot 5.14.0-417.el9.x86_64 2024-02-06T21:22:58.505 DEBUG:teuthology.orchestra.run.smithi135:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-06T21:22:59.179 INFO:teuthology.orchestra.run.smithi135.stderr:Generating grub configuration file ... 2024-02-06T21:23:00.534 INFO:teuthology.orchestra.run.smithi135.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-06T21:23:00.562 INFO:teuthology.orchestra.run.smithi135.stderr:done 2024-02-06T21:23:00.564 DEBUG:teuthology.orchestra.run.smithi135:> mktemp 2024-02-06T21:23:00.578 INFO:teuthology.orchestra.run.smithi135.stdout:/tmp/tmp.N9kola6LNF 2024-02-06T21:23:00.579 DEBUG:teuthology.orchestra.run.smithi135:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.N9kola6LNF 2024-02-06T21:23:00.647 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0666 /tmp/tmp.N9kola6LNF 2024-02-06T21:23:00.782 DEBUG:teuthology.orchestra.remote:smithi135:/tmp/tmp.N9kola6LNF is 6KB 2024-02-06T21:23:00.801 DEBUG:teuthology.orchestra.run.smithi135:> rm -fr /tmp/tmp.N9kola6LNF 2024-02-06T21:23:00.815 DEBUG:teuthology.orchestra.run.smithi135:> sudo /bin/ls /boot/loader/entries || true 2024-02-06T21:23:00.889 INFO:teuthology.orchestra.run.smithi135.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-06T21:23:00.889 INFO:teuthology.orchestra.run.smithi135.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-06T21:23:00.889 INFO:teuthology.orchestra.run.smithi135.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-06T21:23:00.889 INFO:teuthology.orchestra.run.smithi135.stdout:15949e3c88704300882482eddfd7baae-5.14.0-417.el9.x86_64.conf 2024-02-06T21:23:00.890 DEBUG:teuthology.orchestra.run.smithi135:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-417.el9.x86_64 2024-02-06T21:23:01.050 DEBUG:teuthology.orchestra.run.smithi135:> sudo shutdown -r now 2024-02-06T21:23:04.977 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-core-5.14.0-417.el9.x86_64 4/4 2024-02-06T21:23:04.977 INFO:teuthology.orchestra.run.smithi120.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-06T21:23:04.977 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:23:05.047 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-modules-5.14.0-417.el9.x86_64 4/4 2024-02-06T21:23:05.521 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-5.14.0-417.el9.x86_64 4/4 2024-02-06T21:23:05.521 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : kernel-5.14.0-417.el9.x86_64 1/4 2024-02-06T21:23:05.521 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : kernel-core-5.14.0-417.el9.x86_64 2/4 2024-02-06T21:23:05.521 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : kernel-modules-5.14.0-417.el9.x86_64 3/4 2024-02-06T21:23:05.947 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : kernel-modules-core-5.14.0-417.el9.x86_64 4/4 2024-02-06T21:23:05.947 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:23:05.947 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:23:05.947 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-5.14.0-417.el9.x86_64 2024-02-06T21:23:05.947 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-core-5.14.0-417.el9.x86_64 2024-02-06T21:23:05.947 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-modules-5.14.0-417.el9.x86_64 2024-02-06T21:23:05.947 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-modules-core-5.14.0-417.el9.x86_64 2024-02-06T21:23:05.947 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:23:05.948 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:23:06.472 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q kernel | sort -rV | head -n 1 2024-02-06T21:23:06.642 INFO:teuthology.orchestra.run.smithi120.stdout:kernel-5.14.0-417.el9.x86_64 2024-02-06T21:23:06.642 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-417.el9.x86_64 2024-02-06T21:23:06.642 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-417.el9.x86_64 2024-02-06T21:23:06.642 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-06T21:23:06.642 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-06T21:23:06.642 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-06T21:23:06.642 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi120.front.sepia.ceph.com, path=None, version=distro) 2024-02-06T21:23:06.643 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q kernel | sort -rV | head -n 1 2024-02-06T21:23:06.750 INFO:teuthology.orchestra.run.smithi120.stdout:kernel-5.14.0-417.el9.x86_64 2024-02-06T21:23:06.751 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-417.el9.x86_64 2024-02-06T21:23:06.751 DEBUG:teuthology.orchestra.run.smithi120:> sudo rpm -qi grub2-tools 2024-02-06T21:23:06.792 INFO:teuthology.orchestra.run.smithi120.stdout:Name : grub2-tools 2024-02-06T21:23:06.792 INFO:teuthology.orchestra.run.smithi120.stdout:Epoch : 1 2024-02-06T21:23:06.792 INFO:teuthology.orchestra.run.smithi120.stdout:Version : 2.06 2024-02-06T21:23:06.792 INFO:teuthology.orchestra.run.smithi120.stdout:Release : 68.el9 2024-02-06T21:23:06.792 INFO:teuthology.orchestra.run.smithi120.stdout:Architecture: x86_64 2024-02-06T21:23:06.792 INFO:teuthology.orchestra.run.smithi120.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-06T21:23:06.792 INFO:teuthology.orchestra.run.smithi120.stdout:Group : Unspecified 2024-02-06T21:23:06.793 INFO:teuthology.orchestra.run.smithi120.stdout:Size : 8264254 2024-02-06T21:23:06.793 INFO:teuthology.orchestra.run.smithi120.stdout:License : GPLv3+ 2024-02-06T21:23:06.793 INFO:teuthology.orchestra.run.smithi120.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-06T21:23:06.793 INFO:teuthology.orchestra.run.smithi120.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-06T21:23:06.793 INFO:teuthology.orchestra.run.smithi120.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-06T21:23:06.793 INFO:teuthology.orchestra.run.smithi120.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-06T21:23:06.793 INFO:teuthology.orchestra.run.smithi120.stdout:Packager : builder@centos.org 2024-02-06T21:23:06.793 INFO:teuthology.orchestra.run.smithi120.stdout:Vendor : CentOS 2024-02-06T21:23:06.793 INFO:teuthology.orchestra.run.smithi120.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-06T21:23:06.793 INFO:teuthology.orchestra.run.smithi120.stdout:Summary : Support tools for GRUB. 2024-02-06T21:23:06.794 INFO:teuthology.orchestra.run.smithi120.stdout:Description : 2024-02-06T21:23:06.794 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:23:06.794 INFO:teuthology.orchestra.run.smithi120.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-06T21:23:06.794 INFO:teuthology.orchestra.run.smithi120.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-06T21:23:06.794 INFO:teuthology.orchestra.run.smithi120.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-06T21:23:06.794 INFO:teuthology.orchestra.run.smithi120.stdout:hardware devices. 2024-02-06T21:23:06.794 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:23:06.794 INFO:teuthology.orchestra.run.smithi120.stdout:This subpackage provides tools for support of all platforms. 2024-02-06T21:23:06.795 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-06T21:23:06.795 INFO:teuthology.task.kernel:Updating grub on smithi120 to boot 5.14.0-417.el9.x86_64 2024-02-06T21:23:06.795 DEBUG:teuthology.orchestra.run.smithi120:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-06T21:23:07.413 INFO:teuthology.orchestra.run.smithi120.stderr:Generating grub configuration file ... 2024-02-06T21:23:08.773 INFO:teuthology.orchestra.run.smithi120.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-06T21:23:08.803 INFO:teuthology.orchestra.run.smithi120.stderr:done 2024-02-06T21:23:08.805 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2024-02-06T21:23:08.820 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.M2vgFO4dse 2024-02-06T21:23:08.820 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.M2vgFO4dse 2024-02-06T21:23:08.889 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.M2vgFO4dse 2024-02-06T21:23:09.032 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.M2vgFO4dse is 6KB 2024-02-06T21:23:09.081 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.M2vgFO4dse 2024-02-06T21:23:09.095 DEBUG:teuthology.orchestra.run.smithi120:> sudo /bin/ls /boot/loader/entries || true 2024-02-06T21:23:09.160 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-06T21:23:09.160 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-06T21:23:09.160 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-06T21:23:09.160 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-5.14.0-417.el9.x86_64.conf 2024-02-06T21:23:09.161 DEBUG:teuthology.orchestra.run.smithi120:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-417.el9.x86_64 2024-02-06T21:23:09.312 DEBUG:teuthology.orchestra.run.smithi120:> sudo shutdown -r now 2024-02-06T21:23:31.070 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-06T21:23:31.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-02-06T21:23:31.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T21:23:39.322 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-06T21:23:39.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-02-06T21:23:39.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T21:24:31.127 DEBUG:teuthology.orchestra.remote:timed out 2024-02-06T21:24:39.333 DEBUG:teuthology.orchestra.remote:timed out 2024-02-06T21:24:40.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-02-06T21:24:40.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T21:24:41.649 DEBUG:teuthology.orchestra.run.smithi135:> true 2024-02-06T21:24:42.205 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-02-06T21:24:42.205 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-417.el9.x86_64"... 2024-02-06T21:24:42.205 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2024-02-06T21:24:42.219 INFO:teuthology.orchestra.run.smithi135.stdout:5.14.0-417.el9.x86_64 2024-02-06T21:24:42.219 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-417.el9.x86_64 vs 5.14.0-417.el9.x86_64 2024-02-06T21:24:42.219 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-06T21:24:42.220 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-06T21:24:43.220 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-06T21:24:43.221 DEBUG:teuthology.orchestra.run.smithi135:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-06T21:24:43.276 INFO:teuthology.orchestra.run.smithi135.stdout:ttyS1 2024-02-06T21:24:43.326 DEBUG:teuthology.parallel:result is None 2024-02-06T21:24:48.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-02-06T21:24:48.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T21:24:51.647 DEBUG:teuthology.orchestra.run.smithi120:> true 2024-02-06T21:24:52.274 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-02-06T21:24:52.274 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-417.el9.x86_64"... 2024-02-06T21:24:52.275 DEBUG:teuthology.orchestra.run.smithi120:> uname -r 2024-02-06T21:24:52.289 INFO:teuthology.orchestra.run.smithi120.stdout:5.14.0-417.el9.x86_64 2024-02-06T21:24:52.289 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-417.el9.x86_64 vs 5.14.0-417.el9.x86_64 2024-02-06T21:24:52.289 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-06T21:24:52.289 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-06T21:24:53.290 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-06T21:24:53.290 DEBUG:teuthology.orchestra.run.smithi120:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-06T21:24:53.361 INFO:teuthology.orchestra.run.smithi120.stdout:ttyS1 2024-02-06T21:24:53.419 DEBUG:teuthology.parallel:result is None 2024-02-06T21:24:53.419 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-06T21:24:53.427 INFO:teuthology.task.internal:Creating test directory... 2024-02-06T21:24:53.427 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-06T21:24:53.429 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-06T21:24:53.448 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-06T21:24:53.465 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-06T21:24:53.495 INFO:teuthology.task.internal:Creating archive directory... 2024-02-06T21:24:53.495 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-06T21:24:53.497 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-06T21:24:53.628 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-06T21:24:53.635 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-06T21:24:53.635 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-06T21:24:53.637 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-06T21:24:53.666 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-06T21:24:53.675 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-06T21:24:53.680 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-06T21:24:53.688 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-06T21:24:53.690 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-06T21:24:53.697 INFO:teuthology.task.internal:Configuring sudo... 2024-02-06T21:24:53.697 DEBUG:teuthology.orchestra.run.smithi120:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-06T21:24:53.724 DEBUG:teuthology.orchestra.run.smithi135:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-06T21:24:53.761 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-06T21:24:53.770 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-06T21:24:53.770 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-06T21:24:53.790 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-06T21:24:53.817 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-06T21:24:53.887 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-06T21:24:53.934 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-06T21:24:54.014 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-06T21:24:54.057 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:24:54.058 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-06T21:24:54.130 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-06T21:24:54.176 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-06T21:24:54.226 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-06T21:24:54.309 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-06T21:24:54.352 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:24:54.353 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-06T21:24:54.421 DEBUG:teuthology.orchestra.run.smithi120:> sudo service rsyslog restart 2024-02-06T21:24:54.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo service rsyslog restart 2024-02-06T21:24:54.473 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-06T21:24:54.519 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-06T21:24:54.876 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-06T21:24:54.883 INFO:teuthology.task.internal:Starting timer... 2024-02-06T21:24:54.884 INFO:teuthology.run_tasks:Running task pcp... 2024-02-06T21:24:54.893 INFO:teuthology.run_tasks:Running task selinux... 2024-02-06T21:24:54.901 DEBUG:teuthology.orchestra.run.smithi120:> sudo service auditd rotate 2024-02-06T21:24:54.953 INFO:teuthology.orchestra.run.smithi120.stdout:Rotating logs: 2024-02-06T21:24:54.955 DEBUG:teuthology.orchestra.run.smithi135:> sudo service auditd rotate 2024-02-06T21:24:55.039 INFO:teuthology.orchestra.run.smithi135.stdout:Rotating logs: 2024-02-06T21:24:55.041 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-06T21:24:55.041 DEBUG:teuthology.orchestra.run.smithi120:> /usr/sbin/getenforce 2024-02-06T21:24:55.058 INFO:teuthology.orchestra.run.smithi120.stdout:Permissive 2024-02-06T21:24:55.058 DEBUG:teuthology.orchestra.run.smithi135:> /usr/sbin/getenforce 2024-02-06T21:24:55.100 INFO:teuthology.orchestra.run.smithi135.stdout:Permissive 2024-02-06T21:24:55.100 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi120.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi135.front.sepia.ceph.com': 'permissive'} 2024-02-06T21:24:55.100 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-06T21:24:55.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-06T21:24:55.128 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-06T21:24:55.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-06T21:24:55.169 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-06T21:24:55.169 DEBUG:teuthology.orchestra.run.smithi120:> sudo /usr/sbin/setenforce permissive 2024-02-06T21:24:55.201 DEBUG:teuthology.orchestra.run.smithi135:> sudo /usr/sbin/setenforce permissive 2024-02-06T21:24:55.253 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-06T21:24:55.263 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-06T21:24:55.283 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-06T21:24:55.285 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi120.front.sepia.ceph.com,smithi135.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-06T21:29:02.480 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi120.front.sepia.ceph.com'), Remote(name='ubuntu@smithi135.front.sepia.ceph.com')] 2024-02-06T21:29:02.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-02-06T21:29:02.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T21:29:02.575 DEBUG:teuthology.orchestra.run.smithi120:> true 2024-02-06T21:29:02.654 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-02-06T21:29:02.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-02-06T21:29:02.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-06T21:29:02.744 DEBUG:teuthology.orchestra.run.smithi135:> true 2024-02-06T21:29:02.815 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-02-06T21:29:02.816 INFO:teuthology.run_tasks:Running task clock... 2024-02-06T21:29:02.827 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-06T21:29:02.827 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-06T21:29:02.828 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-06T21:29:02.830 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-06T21:29:02.830 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-06T21:29:02.859 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-06T21:29:02.875 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-06T21:29:02.886 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-06T21:29:02.903 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-06T21:29:02.904 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: ntpd: command not found 2024-02-06T21:29:02.918 INFO:teuthology.orchestra.run.smithi120.stdout:506 Cannot talk to daemon 2024-02-06T21:29:02.932 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: ntpd: command not found 2024-02-06T21:29:02.934 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-06T21:29:02.946 INFO:teuthology.orchestra.run.smithi135.stdout:506 Cannot talk to daemon 2024-02-06T21:29:02.950 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-06T21:29:02.962 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-06T21:29:02.978 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-06T21:29:03.012 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-02-06T21:29:03.015 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-06T21:29:03.015 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-02-06T21:29:03.015 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-06T21:29:03.015 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-06T21:29:03.015 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-06T21:29:03.015 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-06T21:29:03.027 INFO:teuthology.orchestra.run.smithi135.stderr:bash: line 1: ntpq: command not found 2024-02-06T21:29:03.029 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-06T21:29:03.029 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2024-02-06T21:29:03.030 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-06T21:29:03.030 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-06T21:29:03.030 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-06T21:29:03.030 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-06T21:29:03.031 INFO:teuthology.run_tasks:Running task install... 2024-02-06T21:29:03.040 DEBUG:teuthology.task.install:project ceph 2024-02-06T21:29:03.040 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '633ab857b9926af935a3e6291c3e1d9251aca357'}} 2024-02-06T21:29:03.040 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '633ab857b9926af935a3e6291c3e1d9251aca357'} 2024-02-06T21:29:03.040 INFO:teuthology.task.install:Using flavor: default 2024-02-06T21:29:03.046 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-02-06T21:29:03.046 INFO:teuthology.task.install:extra packages: [] 2024-02-06T21:29:03.046 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '633ab857b9926af935a3e6291c3e1d9251aca357', 'tag': None, 'wait_for_package': False} 2024-02-06T21:29:03.047 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=633ab857b9926af935a3e6291c3e1d9251aca357 2024-02-06T21:29:03.049 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '633ab857b9926af935a3e6291c3e1d9251aca357', 'tag': None, 'wait_for_package': False} 2024-02-06T21:29:03.049 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=633ab857b9926af935a3e6291c3e1d9251aca357 2024-02-06T21:29:03.271 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-02-05-0849/633ab857b9926af935a3e6291c3e1d9251aca357/centos/9/flavors/default/ 2024-02-06T21:29:03.271 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1269.g633ab857 2024-02-06T21:29:03.284 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-02-05-0849/633ab857b9926af935a3e6291c3e1d9251aca357/centos/9/flavors/default/ 2024-02-06T21:29:03.284 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1269.g633ab857 2024-02-06T21:29:03.394 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-02-05-0849/633ab857b9926af935a3e6291c3e1d9251aca357/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-02-05-0849/633ab857b9926af935a3e6291c3e1d9251aca357/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-02-05-0849/633ab857b9926af935a3e6291c3e1d9251aca357/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-06T21:29:03.394 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:29:03.395 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-06T21:29:03.402 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-02-05-0849/633ab857b9926af935a3e6291c3e1d9251aca357/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-02-05-0849/633ab857b9926af935a3e6291c3e1d9251aca357/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-02-05-0849/633ab857b9926af935a3e6291c3e1d9251aca357/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-06T21:29:03.403 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:29:03.403 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-06T21:29:03.422 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-02-06T21:29:03.423 DEBUG:teuthology.orchestra.run.smithi120:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/633ab857b9926af935a3e6291c3e1d9251aca357/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-06T21:29:03.431 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-02-06T21:29:03.431 DEBUG:teuthology.orchestra.run.smithi135:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/633ab857b9926af935a3e6291c3e1d9251aca357/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-06T21:29:03.490 DEBUG:teuthology.orchestra.run.smithi120:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-06T21:29:03.498 DEBUG:teuthology.orchestra.run.smithi135:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-06T21:29:03.570 DEBUG:teuthology.orchestra.run.smithi120:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-06T21:29:03.578 DEBUG:teuthology.orchestra.run.smithi135:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-06T21:29:03.636 INFO:teuthology.orchestra.run.smithi120.stdout:check_obsoletes = 1 2024-02-06T21:29:03.638 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum clean all 2024-02-06T21:29:03.645 INFO:teuthology.orchestra.run.smithi135.stdout:check_obsoletes = 1 2024-02-06T21:29:03.646 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum clean all 2024-02-06T21:29:03.876 INFO:teuthology.orchestra.run.smithi120.stdout:74 files removed 2024-02-06T21:29:03.893 INFO:teuthology.orchestra.run.smithi135.stdout:74 files removed 2024-02-06T21:29:03.903 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw 2024-02-06T21:29:03.919 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-radosgw 2024-02-06T21:29:04.645 INFO:teuthology.orchestra.run.smithi135.stdout:ceph packages for x86_64 197 kB/s | 80 kB 00:00 2024-02-06T21:29:04.667 INFO:teuthology.orchestra.run.smithi120.stdout:ceph packages for x86_64 179 kB/s | 80 kB 00:00 2024-02-06T21:29:05.051 INFO:teuthology.orchestra.run.smithi120.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-02-06T21:29:05.071 INFO:teuthology.orchestra.run.smithi135.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-02-06T21:29:05.426 INFO:teuthology.orchestra.run.smithi120.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-02-06T21:29:05.480 INFO:teuthology.orchestra.run.smithi135.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-02-06T21:29:06.065 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS-9 - AppStream 12 MB/s | 6.6 MB 00:00 2024-02-06T21:29:06.387 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - AppStream 7.1 MB/s | 6.6 MB 00:00 2024-02-06T21:29:07.460 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-02-06T21:29:07.771 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-02-06T21:29:08.895 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-02-06T21:29:09.176 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.0 MB 00:01 2024-02-06T21:29:10.746 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-02-06T21:29:14.857 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 5.9 MB 00:00 2024-02-06T21:29:17.147 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 15 kB 00:00 2024-02-06T21:29:19.134 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2024-02-06T21:29:24.040 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-02-06T21:29:26.651 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:29:26.674 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:29:26.675 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-02-06T21:29:26.675 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:29:26.675 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:29:26.675 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw x86_64 2:19.0.0-1269.g633ab857.el9 ceph 9.7 M 2024-02-06T21:29:26.675 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrading: 2024-02-06T21:29:26.675 INFO:teuthology.orchestra.run.smithi120.stdout: librados2 x86_64 2:19.0.0-1269.g633ab857.el9 ceph 3.4 M 2024-02-06T21:29:26.675 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1 x86_64 2:19.0.0-1269.g633ab857.el9 ceph 3.1 M 2024-02-06T21:29:26.675 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-02-06T21:29:26.675 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options 2024-02-06T21:29:26.675 INFO:teuthology.orchestra.run.smithi120.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-06T21:29:26.675 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base x86_64 2:19.0.0-1269.g633ab857.el9 ceph 5.1 M 2024-02-06T21:29:26.675 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common x86_64 2:19.0.0-1269.g633ab857.el9 ceph 20 M 2024-02-06T21:29:26.675 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux x86_64 2:19.0.0-1269.g633ab857.el9 ceph 25 k 2024-02-06T21:29:26.676 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-06T21:29:26.676 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-06T21:29:26.676 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-06T21:29:26.676 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2 x86_64 2:19.0.0-1269.g633ab857.el9 ceph 741 k 2024-02-06T21:29:26.676 INFO:teuthology.orchestra.run.smithi120.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-06T21:29:26.676 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-06T21:29:26.676 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-06T21:29:26.676 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1 x86_64 2:19.0.0-1269.g633ab857.el9 ceph 499 k 2024-02-06T21:29:26.676 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-06T21:29:26.676 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2 x86_64 2:19.0.0-1269.g633ab857.el9 ceph 5.1 M 2024-02-06T21:29:26.676 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-06T21:29:26.676 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-06T21:29:26.676 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-06T21:29:26.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse 2024-02-06T21:29:26.677 INFO:teuthology.orchestra.run.smithi120.stdout: x86_64 2:19.0.0-1269.g633ab857.el9 ceph 45 k 2024-02-06T21:29:26.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common x86_64 2:19.0.0-1269.g633ab857.el9 ceph 131 k 2024-02-06T21:29:26.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs x86_64 2:19.0.0-1269.g633ab857.el9 ceph 162 k 2024-02-06T21:29:26.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-06T21:29:26.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados x86_64 2:19.0.0-1269.g633ab857.el9 ceph 320 k 2024-02-06T21:29:26.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd x86_64 2:19.0.0-1269.g633ab857.el9 ceph 299 k 2024-02-06T21:29:26.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw x86_64 2:19.0.0-1269.g633ab857.el9 ceph 99 k 2024-02-06T21:29:26.677 INFO:teuthology.orchestra.run.smithi120.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-06T21:29:26.677 INFO:teuthology.orchestra.run.smithi120.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-06T21:29:26.677 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:29:26.677 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:29:26.677 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:29:26.677 INFO:teuthology.orchestra.run.smithi120.stdout:Install 27 Packages 2024-02-06T21:29:26.678 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrade 2 Packages 2024-02-06T21:29:26.678 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:29:26.678 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 57 M 2024-02-06T21:29:26.678 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:29:26.756 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:16 2024-02-06T21:29:27.483 INFO:teuthology.orchestra.run.smithi120.stdout:(1/29): ceph-base-19.0.0-1269.g633ab857.el9.x86 8.2 MB/s | 5.1 MB 00:00 2024-02-06T21:29:27.524 INFO:teuthology.orchestra.run.smithi120.stdout:(2/29): ceph-selinux-19.0.0-1269.g633ab857.el9. 605 kB/s | 25 kB 00:00 2024-02-06T21:29:27.641 INFO:teuthology.orchestra.run.smithi120.stdout:(3/29): libcephfs2-19.0.0-1269.g633ab857.el9.x8 6.2 MB/s | 741 kB 00:00 2024-02-06T21:29:27.725 INFO:teuthology.orchestra.run.smithi120.stdout:(4/29): libradosstriper1-19.0.0-1269.g633ab857. 5.9 MB/s | 499 kB 00:00 2024-02-06T21:29:30.631 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - CRB 15 MB/s | 5.9 MB 00:00 2024-02-06T21:29:32.449 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2024-02-06T21:29:34.286 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2024-02-06T21:29:34.816 INFO:teuthology.orchestra.run.smithi120.stdout:(5/29): librgw2-19.0.0-1269.g633ab857.el9.x86_6 737 kB/s | 5.1 MB 00:07 2024-02-06T21:29:34.867 INFO:teuthology.orchestra.run.smithi120.stdout:(6/29): python3-ceph-argparse-19.0.0-1269.g633a 908 kB/s | 45 kB 00:00 2024-02-06T21:29:34.917 INFO:teuthology.orchestra.run.smithi120.stdout:(7/29): python3-ceph-common-19.0.0-1269.g633ab8 2.6 MB/s | 131 kB 00:00 2024-02-06T21:29:34.984 INFO:teuthology.orchestra.run.smithi120.stdout:(8/29): python3-cephfs-19.0.0-1269.g633ab857.el 2.4 MB/s | 162 kB 00:00 2024-02-06T21:29:35.042 INFO:teuthology.orchestra.run.smithi120.stdout:(9/29): python3-rados-19.0.0-1269.g633ab857.el9 5.4 MB/s | 320 kB 00:00 2024-02-06T21:29:35.103 INFO:teuthology.orchestra.run.smithi120.stdout:(10/29): python3-rbd-19.0.0-1269.g633ab857.el9. 5.0 MB/s | 299 kB 00:00 2024-02-06T21:29:35.193 INFO:teuthology.orchestra.run.smithi120.stdout:(11/29): python3-rgw-19.0.0-1269.g633ab857.el9. 1.1 MB/s | 99 kB 00:00 2024-02-06T21:29:35.335 INFO:teuthology.orchestra.run.smithi120.stdout:(12/29): ceph-radosgw-19.0.0-1269.g633ab857.el9 1.1 MB/s | 9.7 MB 00:08 2024-02-06T21:29:35.401 INFO:teuthology.orchestra.run.smithi120.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2024-02-06T21:29:35.427 INFO:teuthology.orchestra.run.smithi120.stdout:(14/29): boost-program-options-1.75.0-8.el9.x86 453 kB/s | 106 kB 00:00 2024-02-06T21:29:35.444 INFO:teuthology.orchestra.run.smithi120.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-02-06T21:29:35.494 INFO:teuthology.orchestra.run.smithi120.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-02-06T21:29:35.519 INFO:teuthology.orchestra.run.smithi120.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-02-06T21:29:35.536 INFO:teuthology.orchestra.run.smithi120.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 999 kB/s | 41 kB 00:00 2024-02-06T21:29:35.862 INFO:teuthology.orchestra.run.smithi120.stdout:(19/29): ceph-common-19.0.0-1269.g633ab857.el9. 2.2 MB/s | 20 MB 00:09 2024-02-06T21:29:35.970 INFO:teuthology.orchestra.run.smithi120.stdout:(20/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 686 kB/s | 309 kB 00:00 2024-02-06T21:29:36.003 INFO:teuthology.orchestra.run.smithi120.stdout:(21/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 184 kB/s | 26 kB 00:00 2024-02-06T21:29:36.029 INFO:teuthology.orchestra.run.smithi120.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 830 kB/s | 49 kB 00:00 2024-02-06T21:29:36.112 INFO:teuthology.orchestra.run.smithi120.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 621 kB/s | 67 kB 00:00 2024-02-06T21:29:36.180 INFO:teuthology.orchestra.run.smithi120.stdout:(24/29): libarrow-9.0.0-10.el9.x86_64.rpm 6.9 MB/s | 4.4 MB 00:00 2024-02-06T21:29:36.204 INFO:teuthology.orchestra.run.smithi120.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-02-06T21:29:36.247 INFO:teuthology.orchestra.run.smithi120.stdout:(26/29): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-02-06T21:29:36.288 INFO:teuthology.orchestra.run.smithi120.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-02-06T21:29:36.564 INFO:teuthology.orchestra.run.smithi120.stdout:(28/29): librbd1-19.0.0-1269.g633ab857.el9.x86_ 9.9 MB/s | 3.1 MB 00:00 2024-02-06T21:29:36.614 INFO:teuthology.orchestra.run.smithi120.stdout:(29/29): librados2-19.0.0-1269.g633ab857.el9.x8 8.3 MB/s | 3.4 MB 00:00 2024-02-06T21:29:36.617 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:29:36.618 INFO:teuthology.orchestra.run.smithi120.stdout:Total 5.8 MB/s | 57 MB 00:09 2024-02-06T21:29:36.618 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:29:36.654 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:29:36.654 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:29:37.114 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:29:37.115 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:29:38.509 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:29:38.596 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-02-06T21:29:38.627 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-02-06T21:29:38.680 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-02-06T21:29:39.004 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-02-06T21:29:39.059 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librados2-2:19.0.0-1269.g633ab857.el9.x86_64 5/31 2024-02-06T21:29:39.134 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-02-06T21:29:39.151 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:19.0.0-1269.g633ab857.el9.x86_64 5/31 2024-02-06T21:29:39.254 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rados-2:19.0.0-1269.g633ab857.el9.x86_64 6/31 2024-02-06T21:29:39.286 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs2-2:19.0.0-1269.g633ab857.el9.x86_64 7/31 2024-02-06T21:29:39.331 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs2-2:19.0.0-1269.g633ab857.el9.x86_64 7/31 2024-02-06T21:29:39.360 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-02-06T21:29:39.403 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-argparse-2:19.0.0-1269.g633ab857.el9. 9/31 2024-02-06T21:29:39.472 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cephfs-2:19.0.0-1269.g633ab857.el9.x86_64 10/31 2024-02-06T21:29:39.496 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libradosstriper1-2:19.0.0-1269.g633ab857.el9.x86_6 11/31 2024-02-06T21:29:39.561 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1269.g633ab857.el9.x86_6 11/31 2024-02-06T21:29:39.669 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-02-06T21:29:39.704 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-02-06T21:29:39.743 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-02-06T21:29:39.777 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-02-06T21:29:39.805 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-02-06T21:29:39.838 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-02-06T21:29:40.155 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-02-06T21:29:40.184 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librbd1-2:19.0.0-1269.g633ab857.el9.x86_64 19/31 2024-02-06T21:29:40.274 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:19.0.0-1269.g633ab857.el9.x86_64 19/31 2024-02-06T21:29:40.306 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rbd-2:19.0.0-1269.g633ab857.el9.x86_64 20/31 2024-02-06T21:29:40.348 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-02-06T21:29:40.797 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-02-06T21:29:40.828 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librgw2-2:19.0.0-1269.g633ab857.el9.x86_64 23/31 2024-02-06T21:29:40.883 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librgw2-2:19.0.0-1269.g633ab857.el9.x86_64 23/31 2024-02-06T21:29:40.950 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rgw-2:19.0.0-1269.g633ab857.el9.x86_64 24/31 2024-02-06T21:29:40.981 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-common-2:19.0.0-1269.g633ab857.el9.x8 25/31 2024-02-06T21:29:41.711 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:29:41.733 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:29:41.733 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-02-06T21:29:41.733 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:29:41.733 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:29:41.733 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw x86_64 2:19.0.0-1269.g633ab857.el9 ceph 9.7 M 2024-02-06T21:29:41.733 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrading: 2024-02-06T21:29:41.733 INFO:teuthology.orchestra.run.smithi135.stdout: librados2 x86_64 2:19.0.0-1269.g633ab857.el9 ceph 3.4 M 2024-02-06T21:29:41.733 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1 x86_64 2:19.0.0-1269.g633ab857.el9 ceph 3.1 M 2024-02-06T21:29:41.733 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-02-06T21:29:41.733 INFO:teuthology.orchestra.run.smithi135.stdout: boost-program-options 2024-02-06T21:29:41.734 INFO:teuthology.orchestra.run.smithi135.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-06T21:29:41.734 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base x86_64 2:19.0.0-1269.g633ab857.el9 ceph 5.1 M 2024-02-06T21:29:41.734 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common x86_64 2:19.0.0-1269.g633ab857.el9 ceph 20 M 2024-02-06T21:29:41.734 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux x86_64 2:19.0.0-1269.g633ab857.el9 ceph 25 k 2024-02-06T21:29:41.734 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-06T21:29:41.734 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-06T21:29:41.734 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-06T21:29:41.734 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2 x86_64 2:19.0.0-1269.g633ab857.el9 ceph 741 k 2024-02-06T21:29:41.734 INFO:teuthology.orchestra.run.smithi135.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-06T21:29:41.734 INFO:teuthology.orchestra.run.smithi135.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-06T21:29:41.734 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-06T21:29:41.734 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1 x86_64 2:19.0.0-1269.g633ab857.el9 ceph 499 k 2024-02-06T21:29:41.734 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-06T21:29:41.734 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2 x86_64 2:19.0.0-1269.g633ab857.el9 ceph 5.1 M 2024-02-06T21:29:41.735 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-06T21:29:41.735 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-06T21:29:41.735 INFO:teuthology.orchestra.run.smithi135.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-06T21:29:41.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse 2024-02-06T21:29:41.735 INFO:teuthology.orchestra.run.smithi135.stdout: x86_64 2:19.0.0-1269.g633ab857.el9 ceph 45 k 2024-02-06T21:29:41.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common x86_64 2:19.0.0-1269.g633ab857.el9 ceph 131 k 2024-02-06T21:29:41.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs x86_64 2:19.0.0-1269.g633ab857.el9 ceph 162 k 2024-02-06T21:29:41.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-06T21:29:41.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados x86_64 2:19.0.0-1269.g633ab857.el9 ceph 320 k 2024-02-06T21:29:41.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd x86_64 2:19.0.0-1269.g633ab857.el9 ceph 299 k 2024-02-06T21:29:41.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw x86_64 2:19.0.0-1269.g633ab857.el9 ceph 99 k 2024-02-06T21:29:41.735 INFO:teuthology.orchestra.run.smithi135.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-06T21:29:41.735 INFO:teuthology.orchestra.run.smithi135.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-06T21:29:41.735 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:29:41.735 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:29:41.736 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:29:41.736 INFO:teuthology.orchestra.run.smithi135.stdout:Install 27 Packages 2024-02-06T21:29:41.736 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrade 2 Packages 2024-02-06T21:29:41.736 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:29:41.736 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 57 M 2024-02-06T21:29:41.736 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:29:42.560 INFO:teuthology.orchestra.run.smithi135.stdout:(1/29): ceph-radosgw-19.0.0-1269.g633ab857.el9. 12 MB/s | 9.7 MB 00:00 2024-02-06T21:29:42.610 INFO:teuthology.orchestra.run.smithi135.stdout:(2/29): ceph-selinux-19.0.0-1269.g633ab857.el9. 505 kB/s | 25 kB 00:00 2024-02-06T21:29:42.693 INFO:teuthology.orchestra.run.smithi135.stdout:(3/29): libcephfs2-19.0.0-1269.g633ab857.el9.x8 8.7 MB/s | 741 kB 00:00 2024-02-06T21:29:42.752 INFO:teuthology.orchestra.run.smithi135.stdout:(4/29): libradosstriper1-19.0.0-1269.g633ab857. 8.4 MB/s | 499 kB 00:00 2024-02-06T21:29:43.086 INFO:teuthology.orchestra.run.smithi135.stdout:(5/29): librgw2-19.0.0-1269.g633ab857.el9.x86_6 15 MB/s | 5.1 MB 00:00 2024-02-06T21:29:43.144 INFO:teuthology.orchestra.run.smithi135.stdout:(6/29): python3-ceph-argparse-19.0.0-1269.g633a 776 kB/s | 45 kB 00:00 2024-02-06T21:29:43.174 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:19.0.0-1269.g633ab857.el9.x86_64 26/31 2024-02-06T21:29:43.219 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-common-2:19.0.0-1269.g633ab857.el9.x86_64 26/31 2024-02-06T21:29:43.253 INFO:teuthology.orchestra.run.smithi135.stdout:(7/29): ceph-base-19.0.0-1269.g633ab857.el9.x86 3.5 MB/s | 5.1 MB 00:01 2024-02-06T21:29:43.286 INFO:teuthology.orchestra.run.smithi135.stdout:(8/29): python3-ceph-common-19.0.0-1269.g633ab8 925 kB/s | 131 kB 00:00 2024-02-06T21:29:43.312 INFO:teuthology.orchestra.run.smithi135.stdout:(9/29): python3-cephfs-19.0.0-1269.g633ab857.el 2.7 MB/s | 162 kB 00:00 2024-02-06T21:29:43.354 INFO:teuthology.orchestra.run.smithi135.stdout:(10/29): python3-rados-19.0.0-1269.g633ab857.el 4.7 MB/s | 320 kB 00:00 2024-02-06T21:29:43.379 INFO:teuthology.orchestra.run.smithi135.stdout:(11/29): python3-rbd-19.0.0-1269.g633ab857.el9. 4.4 MB/s | 299 kB 00:00 2024-02-06T21:29:43.404 INFO:teuthology.orchestra.run.smithi135.stdout:(12/29): python3-rgw-19.0.0-1269.g633ab857.el9. 1.9 MB/s | 99 kB 00:00 2024-02-06T21:29:43.521 INFO:teuthology.orchestra.run.smithi135.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 746 kB/s | 106 kB 00:00 2024-02-06T21:29:43.546 INFO:teuthology.orchestra.run.smithi135.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-02-06T21:29:43.563 INFO:teuthology.orchestra.run.smithi135.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-02-06T21:29:43.622 INFO:teuthology.orchestra.run.smithi135.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-02-06T21:29:43.656 INFO:teuthology.orchestra.run.smithi135.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-02-06T21:29:43.673 INFO:teuthology.orchestra.run.smithi135.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 821 kB/s | 41 kB 00:00 2024-02-06T21:29:43.762 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:19.0.0-1269.g633ab857.el9.x86_64 26/31 2024-02-06T21:29:43.797 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-base-2:19.0.0-1269.g633ab857.el9.x86_64 27/31 2024-02-06T21:29:43.873 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-base-2:19.0.0-1269.g633ab857.el9.x86_64 27/31 2024-02-06T21:29:43.873 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-06T21:29:43.873 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-06T21:29:43.873 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:29:43.915 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-selinux-2:19.0.0-1269.g633ab857.el9.x86_64 28/31 2024-02-06T21:29:44.148 INFO:teuthology.orchestra.run.smithi135.stdout:(19/29): ceph-common-19.0.0-1269.g633ab857.el9. 8.4 MB/s | 20 MB 00:02 2024-02-06T21:29:44.315 INFO:teuthology.orchestra.run.smithi135.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 156 kB/s | 26 kB 00:00 2024-02-06T21:29:44.407 INFO:teuthology.orchestra.run.smithi135.stdout:(21/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 411 kB/s | 309 kB 00:00 2024-02-06T21:29:44.424 INFO:teuthology.orchestra.run.smithi135.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-02-06T21:29:44.491 INFO:teuthology.orchestra.run.smithi135.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 811 kB/s | 67 kB 00:00 2024-02-06T21:29:44.766 INFO:teuthology.orchestra.run.smithi135.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 697 kB/s | 192 kB 00:00 2024-02-06T21:29:44.950 INFO:teuthology.orchestra.run.smithi135.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.6 MB/s | 838 kB 00:00 2024-02-06T21:29:45.483 INFO:teuthology.orchestra.run.smithi135.stdout:(26/29): librados2-19.0.0-1269.g633ab857.el9.x8 6.3 MB/s | 3.4 MB 00:00 2024-02-06T21:29:45.525 INFO:teuthology.orchestra.run.smithi135.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-02-06T21:29:45.800 INFO:teuthology.orchestra.run.smithi135.stdout:(28/29): librbd1-19.0.0-1269.g633ab857.el9.x86_ 9.9 MB/s | 3.1 MB 00:00 2024-02-06T21:29:46.650 INFO:teuthology.orchestra.run.smithi135.stdout:(29/29): libarrow-9.0.0-10.el9.x86_64.rpm 1.5 MB/s | 4.4 MB 00:02 2024-02-06T21:29:46.656 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:29:46.657 INFO:teuthology.orchestra.run.smithi135.stdout:Total 12 MB/s | 57 MB 00:04 2024-02-06T21:29:46.657 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:29:46.692 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:29:46.693 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:29:47.165 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:29:47.165 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:29:48.523 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:29:48.603 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-02-06T21:29:48.638 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-02-06T21:29:48.688 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-02-06T21:29:49.006 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-02-06T21:29:49.071 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librados2-2:19.0.0-1269.g633ab857.el9.x86_64 5/31 2024-02-06T21:29:49.162 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-2:19.0.0-1269.g633ab857.el9.x86_64 5/31 2024-02-06T21:29:49.248 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rados-2:19.0.0-1269.g633ab857.el9.x86_64 6/31 2024-02-06T21:29:49.281 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs2-2:19.0.0-1269.g633ab857.el9.x86_64 7/31 2024-02-06T21:29:49.343 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs2-2:19.0.0-1269.g633ab857.el9.x86_64 7/31 2024-02-06T21:29:49.371 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-02-06T21:29:49.415 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-argparse-2:19.0.0-1269.g633ab857.el9. 9/31 2024-02-06T21:29:49.484 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cephfs-2:19.0.0-1269.g633ab857.el9.x86_64 10/31 2024-02-06T21:29:49.508 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libradosstriper1-2:19.0.0-1269.g633ab857.el9.x86_6 11/31 2024-02-06T21:29:49.556 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1269.g633ab857.el9.x86_6 11/31 2024-02-06T21:29:49.663 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-02-06T21:29:49.707 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-02-06T21:29:49.747 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-02-06T21:29:49.788 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-02-06T21:29:49.816 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-02-06T21:29:49.850 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-02-06T21:29:50.167 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-02-06T21:29:50.196 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librbd1-2:19.0.0-1269.g633ab857.el9.x86_64 19/31 2024-02-06T21:29:50.285 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-2:19.0.0-1269.g633ab857.el9.x86_64 19/31 2024-02-06T21:29:50.317 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rbd-2:19.0.0-1269.g633ab857.el9.x86_64 20/31 2024-02-06T21:29:50.363 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-02-06T21:29:50.817 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-02-06T21:29:50.840 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librgw2-2:19.0.0-1269.g633ab857.el9.x86_64 23/31 2024-02-06T21:29:50.911 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librgw2-2:19.0.0-1269.g633ab857.el9.x86_64 23/31 2024-02-06T21:29:50.978 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rgw-2:19.0.0-1269.g633ab857.el9.x86_64 24/31 2024-02-06T21:29:51.009 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-common-2:19.0.0-1269.g633ab857.el9.x8 25/31 2024-02-06T21:29:53.283 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1269.g633ab857.el9.x86_64 28/31 2024-02-06T21:29:53.283 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /sys 2024-02-06T21:29:53.283 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /proc 2024-02-06T21:29:53.283 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /mnt 2024-02-06T21:29:53.283 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /var/tmp 2024-02-06T21:29:53.283 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /home 2024-02-06T21:29:53.283 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /root 2024-02-06T21:29:53.283 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /tmp 2024-02-06T21:29:53.283 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:29:53.287 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:19.0.0-1269.g633ab857.el9.x86_64 26/31 2024-02-06T21:29:53.331 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-common-2:19.0.0-1269.g633ab857.el9.x86_64 26/31 2024-02-06T21:29:53.871 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:19.0.0-1269.g633ab857.el9.x86_64 26/31 2024-02-06T21:29:53.909 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-base-2:19.0.0-1269.g633ab857.el9.x86_64 27/31 2024-02-06T21:29:54.002 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-base-2:19.0.0-1269.g633ab857.el9.x86_64 27/31 2024-02-06T21:29:54.002 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-06T21:29:54.002 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-06T21:29:54.002 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:29:54.034 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-selinux-2:19.0.0-1269.g633ab857.el9.x86_64 28/31 2024-02-06T21:29:54.180 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-radosgw-2:19.0.0-1269.g633ab857.el9.x86_64 29/31 2024-02-06T21:29:54.204 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1269.g633ab857.el9.x86_64 29/31 2024-02-06T21:29:54.204 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:29:54.205 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-06T21:29:54.205 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-06T21:29:54.205 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-06T21:29:54.205 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:29:54.206 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-06T21:29:54.291 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-06T21:29:54.291 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-06T21:29:55.403 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-06T21:29:55.403 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-base-2:19.0.0-1269.g633ab857.el9.x86_64 1/31 2024-02-06T21:29:55.403 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-common-2:19.0.0-1269.g633ab857.el9.x86_64 2/31 2024-02-06T21:29:55.403 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-radosgw-2:19.0.0-1269.g633ab857.el9.x86_64 3/31 2024-02-06T21:29:55.403 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-selinux-2:19.0.0-1269.g633ab857.el9.x86_64 4/31 2024-02-06T21:29:55.403 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs2-2:19.0.0-1269.g633ab857.el9.x86_64 5/31 2024-02-06T21:29:55.403 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libradosstriper1-2:19.0.0-1269.g633ab857.el9.x86_6 6/31 2024-02-06T21:29:55.403 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librgw2-2:19.0.0-1269.g633ab857.el9.x86_64 7/31 2024-02-06T21:29:55.403 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1269.g633ab857.el9. 8/31 2024-02-06T21:29:55.403 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-common-2:19.0.0-1269.g633ab857.el9.x8 9/31 2024-02-06T21:29:55.404 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cephfs-2:19.0.0-1269.g633ab857.el9.x86_64 10/31 2024-02-06T21:29:55.404 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rados-2:19.0.0-1269.g633ab857.el9.x86_64 11/31 2024-02-06T21:29:55.404 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rbd-2:19.0.0-1269.g633ab857.el9.x86_64 12/31 2024-02-06T21:29:55.404 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rgw-2:19.0.0-1269.g633ab857.el9.x86_64 13/31 2024-02-06T21:29:55.404 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-02-06T21:29:55.404 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-02-06T21:29:55.404 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-02-06T21:29:55.404 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-02-06T21:29:55.404 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-02-06T21:29:55.404 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-02-06T21:29:55.404 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-02-06T21:29:55.404 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-02-06T21:29:55.404 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-02-06T21:29:55.405 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-02-06T21:29:55.405 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-02-06T21:29:55.405 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-02-06T21:29:55.405 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-02-06T21:29:55.405 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-02-06T21:29:55.405 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:19.0.0-1269.g633ab857.el9.x86_64 28/31 2024-02-06T21:29:55.405 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-02-06T21:29:55.405 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:19.0.0-1269.g633ab857.el9.x86_64 30/31 2024-02-06T21:29:56.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-02-06T21:29:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:29:56.356 INFO:teuthology.orchestra.run.smithi120.stdout:Upgraded: 2024-02-06T21:29:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:29:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:29:56.356 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:29:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-06T21:29:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:29:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:29:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:29:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:29:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-06T21:29:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-06T21:29:56.356 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-06T21:29:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:29:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-06T21:29:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-06T21:29:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-06T21:29:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:29:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-06T21:29:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:29:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-06T21:29:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-06T21:29:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-06T21:29:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:29:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:29:56.357 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:29:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-06T21:29:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:29:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:29:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:29:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: re2-1:20211101-3.el9.x86_64 2024-02-06T21:29:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-06T21:29:56.358 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:29:56.358 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:29:56.748 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-test 2024-02-06T21:29:57.323 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:00:33 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:29:57.484 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:29:57.485 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:29:57.485 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-02-06T21:29:57.485 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:29:57.485 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:29:57.485 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test x86_64 2:19.0.0-1269.g633ab857.el9 ceph 47 M 2024-02-06T21:29:57.486 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-02-06T21:29:57.486 INFO:teuthology.orchestra.run.smithi120.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-06T21:29:57.486 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite x86_64 2:19.0.0-1269.g633ab857.el9 ceph 163 k 2024-02-06T21:29:57.486 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-06T21:29:57.486 INFO:teuthology.orchestra.run.smithi120.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-06T21:29:57.486 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-06T21:29:57.486 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:29:57.486 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:29:57.486 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:29:57.486 INFO:teuthology.orchestra.run.smithi120.stdout:Install 6 Packages 2024-02-06T21:29:57.486 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:29:57.487 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 48 M 2024-02-06T21:29:57.487 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 204 M 2024-02-06T21:29:57.487 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:29:57.736 INFO:teuthology.orchestra.run.smithi120.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 300 kB 00:00 2024-02-06T21:29:57.777 INFO:teuthology.orchestra.run.smithi120.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-02-06T21:29:57.853 INFO:teuthology.orchestra.run.smithi120.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 2.4 MB/s | 186 kB 00:00 2024-02-06T21:29:58.003 INFO:teuthology.orchestra.run.smithi120.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-02-06T21:29:58.028 INFO:teuthology.orchestra.run.smithi120.stdout:(5/6): libcephsqlite-19.0.0-1269.g633ab857.el9. 302 kB/s | 163 kB 00:00 2024-02-06T21:30:01.787 INFO:teuthology.orchestra.run.smithi120.stdout:(6/6): ceph-test-19.0.0-1269.g633ab857.el9.x86_ 11 MB/s | 47 MB 00:04 2024-02-06T21:30:01.790 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:01.790 INFO:teuthology.orchestra.run.smithi120.stdout:Total 11 MB/s | 48 MB 00:04 2024-02-06T21:30:01.790 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:30:01.810 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:30:01.811 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:30:02.207 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:30:02.207 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:30:02.781 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:30:02.859 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-06T21:30:02.894 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-06T21:30:02.949 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-06T21:30:02.995 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-06T21:30:03.023 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephsqlite-2:19.0.0-1269.g633ab857.el9.x86_64 5/6 2024-02-06T21:30:03.659 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1269.g633ab857.el9.x86_64 28/31 2024-02-06T21:30:03.659 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /sys 2024-02-06T21:30:03.659 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /proc 2024-02-06T21:30:03.659 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /mnt 2024-02-06T21:30:03.659 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /var/tmp 2024-02-06T21:30:03.659 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /home 2024-02-06T21:30:03.659 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /root 2024-02-06T21:30:03.659 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /tmp 2024-02-06T21:30:03.659 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:04.584 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-radosgw-2:19.0.0-1269.g633ab857.el9.x86_64 29/31 2024-02-06T21:30:04.608 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1269.g633ab857.el9.x86_64 29/31 2024-02-06T21:30:04.608 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:30:04.608 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-06T21:30:04.609 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-06T21:30:04.609 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-06T21:30:04.609 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:04.610 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-06T21:30:04.703 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-06T21:30:04.703 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-06T21:30:05.824 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-06T21:30:05.824 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-base-2:19.0.0-1269.g633ab857.el9.x86_64 1/31 2024-02-06T21:30:05.824 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-common-2:19.0.0-1269.g633ab857.el9.x86_64 2/31 2024-02-06T21:30:05.824 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-radosgw-2:19.0.0-1269.g633ab857.el9.x86_64 3/31 2024-02-06T21:30:05.824 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-selinux-2:19.0.0-1269.g633ab857.el9.x86_64 4/31 2024-02-06T21:30:05.824 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs2-2:19.0.0-1269.g633ab857.el9.x86_64 5/31 2024-02-06T21:30:05.824 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libradosstriper1-2:19.0.0-1269.g633ab857.el9.x86_6 6/31 2024-02-06T21:30:05.824 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librgw2-2:19.0.0-1269.g633ab857.el9.x86_64 7/31 2024-02-06T21:30:05.824 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1269.g633ab857.el9. 8/31 2024-02-06T21:30:05.824 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-common-2:19.0.0-1269.g633ab857.el9.x8 9/31 2024-02-06T21:30:05.824 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cephfs-2:19.0.0-1269.g633ab857.el9.x86_64 10/31 2024-02-06T21:30:05.824 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rados-2:19.0.0-1269.g633ab857.el9.x86_64 11/31 2024-02-06T21:30:05.825 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rbd-2:19.0.0-1269.g633ab857.el9.x86_64 12/31 2024-02-06T21:30:05.825 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rgw-2:19.0.0-1269.g633ab857.el9.x86_64 13/31 2024-02-06T21:30:05.825 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-02-06T21:30:05.825 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-02-06T21:30:05.825 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-02-06T21:30:05.825 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-02-06T21:30:05.825 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-02-06T21:30:05.825 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-02-06T21:30:05.825 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-02-06T21:30:05.825 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-02-06T21:30:05.825 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-02-06T21:30:05.825 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-02-06T21:30:05.825 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-02-06T21:30:05.826 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-02-06T21:30:05.826 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-02-06T21:30:05.826 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-02-06T21:30:05.826 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-2:19.0.0-1269.g633ab857.el9.x86_64 28/31 2024-02-06T21:30:05.826 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-02-06T21:30:05.826 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-2:19.0.0-1269.g633ab857.el9.x86_64 30/31 2024-02-06T21:30:06.884 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-02-06T21:30:06.884 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:06.884 INFO:teuthology.orchestra.run.smithi135.stdout:Upgraded: 2024-02-06T21:30:06.884 INFO:teuthology.orchestra.run.smithi135.stdout: librados2-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:06.884 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:06.885 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:30:06.885 INFO:teuthology.orchestra.run.smithi135.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-06T21:30:06.885 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:06.885 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:06.885 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:06.885 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:06.885 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-06T21:30:06.885 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-06T21:30:06.885 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-06T21:30:06.885 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:06.885 INFO:teuthology.orchestra.run.smithi135.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-06T21:30:06.885 INFO:teuthology.orchestra.run.smithi135.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-06T21:30:06.885 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-06T21:30:06.885 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:06.886 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-06T21:30:06.886 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:06.886 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-06T21:30:06.886 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-06T21:30:06.886 INFO:teuthology.orchestra.run.smithi135.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-06T21:30:06.886 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:06.886 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:06.886 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:06.886 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-06T21:30:06.886 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:06.886 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:06.886 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:06.887 INFO:teuthology.orchestra.run.smithi135.stdout: re2-1:20211101-3.el9.x86_64 2024-02-06T21:30:06.887 INFO:teuthology.orchestra.run.smithi135.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-06T21:30:06.887 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:06.887 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:30:07.238 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-test 2024-02-06T21:30:07.530 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1269.g633ab857.el9.x86_64 5/6 2024-02-06T21:30:07.614 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-test-2:19.0.0-1269.g633ab857.el9.x86_64 6/6 2024-02-06T21:30:07.807 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:28 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:30:07.964 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:30:07.965 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:30:07.965 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-02-06T21:30:07.965 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:30:07.965 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:30:07.965 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test x86_64 2:19.0.0-1269.g633ab857.el9 ceph 47 M 2024-02-06T21:30:07.965 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-02-06T21:30:07.965 INFO:teuthology.orchestra.run.smithi135.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-06T21:30:07.965 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite x86_64 2:19.0.0-1269.g633ab857.el9 ceph 163 k 2024-02-06T21:30:07.965 INFO:teuthology.orchestra.run.smithi135.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-06T21:30:07.966 INFO:teuthology.orchestra.run.smithi135.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-06T21:30:07.966 INFO:teuthology.orchestra.run.smithi135.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-06T21:30:07.966 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:07.966 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:30:07.966 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:30:07.966 INFO:teuthology.orchestra.run.smithi135.stdout:Install 6 Packages 2024-02-06T21:30:07.966 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:07.967 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 48 M 2024-02-06T21:30:07.967 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 204 M 2024-02-06T21:30:07.967 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:30:08.139 INFO:teuthology.orchestra.run.smithi135.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 300 kB 00:00 2024-02-06T21:30:08.239 INFO:teuthology.orchestra.run.smithi135.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 626 kB/s | 63 kB 00:00 2024-02-06T21:30:08.290 INFO:teuthology.orchestra.run.smithi135.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-02-06T21:30:08.332 INFO:teuthology.orchestra.run.smithi135.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-02-06T21:30:08.382 INFO:teuthology.orchestra.run.smithi135.stdout:(5/6): libcephsqlite-19.0.0-1269.g633ab857.el9. 394 kB/s | 163 kB 00:00 2024-02-06T21:30:10.081 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-test-2:19.0.0-1269.g633ab857.el9.x86_64 6/6 2024-02-06T21:30:10.081 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-test-2:19.0.0-1269.g633ab857.el9.x86_64 1/6 2024-02-06T21:30:10.081 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephsqlite-2:19.0.0-1269.g633ab857.el9.x86_64 2/6 2024-02-06T21:30:10.081 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-06T21:30:10.081 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-06T21:30:10.082 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-06T21:30:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-06T21:30:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:10.450 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:30:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: jq-1.6-16.el9.x86_64 2024-02-06T21:30:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-06T21:30:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-06T21:30:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-06T21:30:10.450 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:10.450 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:30:10.465 INFO:teuthology.orchestra.run.smithi135.stdout:(6/6): ceph-test-19.0.0-1269.g633ab857.el9.x86_ 19 MB/s | 47 MB 00:02 2024-02-06T21:30:10.468 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:10.468 INFO:teuthology.orchestra.run.smithi135.stdout:Total 19 MB/s | 48 MB 00:02 2024-02-06T21:30:10.469 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:30:10.488 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:30:10.489 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:30:10.715 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph 2024-02-06T21:30:10.878 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:30:10.878 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:30:11.314 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:00:47 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:30:11.454 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:30:11.486 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:30:11.489 INFO:teuthology.orchestra.run.smithi120.stdout:========================================================================================= 2024-02-06T21:30:11.489 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-02-06T21:30:11.489 INFO:teuthology.orchestra.run.smithi120.stdout:========================================================================================= 2024-02-06T21:30:11.489 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:30:11.489 INFO:teuthology.orchestra.run.smithi120.stdout: ceph x86_64 2:19.0.0-1269.g633ab857.el9 ceph 6.6 k 2024-02-06T21:30:11.489 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-02-06T21:30:11.489 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds x86_64 2:19.0.0-1269.g633ab857.el9 ceph 2.1 M 2024-02-06T21:30:11.490 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr x86_64 2:19.0.0-1269.g633ab857.el9 ceph 1.5 M 2024-02-06T21:30:11.490 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 247 k 2024-02-06T21:30:11.490 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon x86_64 2:19.0.0-1269.g633ab857.el9 ceph 4.7 M 2024-02-06T21:30:11.490 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd x86_64 2:19.0.0-1269.g633ab857.el9 ceph 17 M 2024-02-06T21:30:11.490 INFO:teuthology.orchestra.run.smithi120.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-06T21:30:11.490 INFO:teuthology.orchestra.run.smithi120.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-06T21:30:11.490 INFO:teuthology.orchestra.run.smithi120.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-06T21:30:11.490 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-06T21:30:11.490 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-06T21:30:11.490 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-06T21:30:11.490 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-06T21:30:11.490 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-06T21:30:11.491 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-06T21:30:11.491 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-06T21:30:11.491 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-06T21:30:11.491 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-06T21:30:11.491 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-06T21:30:11.491 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-06T21:30:11.491 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-06T21:30:11.491 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-06T21:30:11.491 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-06T21:30:11.491 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-06T21:30:11.491 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-06T21:30:11.491 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-06T21:30:11.492 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-06T21:30:11.492 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-06T21:30:11.492 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-06T21:30:11.492 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-06T21:30:11.492 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-06T21:30:11.492 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-06T21:30:11.492 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-06T21:30:11.492 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-06T21:30:11.492 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-06T21:30:11.492 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-06T21:30:11.492 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:11.493 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:30:11.493 INFO:teuthology.orchestra.run.smithi120.stdout:========================================================================================= 2024-02-06T21:30:11.493 INFO:teuthology.orchestra.run.smithi120.stdout:Install 36 Packages 2024-02-06T21:30:11.493 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:11.493 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 30 M 2024-02-06T21:30:11.494 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 107 M 2024-02-06T21:30:11.494 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:30:11.535 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-06T21:30:11.566 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-06T21:30:11.610 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-06T21:30:11.650 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-06T21:30:11.678 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephsqlite-2:19.0.0-1269.g633ab857.el9.x86_64 5/6 2024-02-06T21:30:11.846 INFO:teuthology.orchestra.run.smithi120.stdout:(1/36): ceph-19.0.0-1269.g633ab857.el9.x86_64.r 38 kB/s | 6.6 kB 00:00 2024-02-06T21:30:12.063 INFO:teuthology.orchestra.run.smithi120.stdout:(2/36): ceph-mds-19.0.0-1269.g633ab857.el9.x86_ 5.4 MB/s | 2.1 MB 00:00 2024-02-06T21:30:12.480 INFO:teuthology.orchestra.run.smithi120.stdout:(3/36): ceph-mgr-19.0.0-1269.g633ab857.el9.x86_ 1.8 MB/s | 1.5 MB 00:00 2024-02-06T21:30:12.714 INFO:teuthology.orchestra.run.smithi120.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1269.g633a 1.0 MB/s | 247 kB 00:00 2024-02-06T21:30:12.956 INFO:teuthology.orchestra.run.smithi120.stdout:(5/36): ceph-osd-19.0.0-1269.g633ab857.el9.x86_ 19 MB/s | 17 MB 00:00 2024-02-06T21:30:12.981 INFO:teuthology.orchestra.run.smithi120.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 699 kB/s | 187 kB 00:00 2024-02-06T21:30:13.006 INFO:teuthology.orchestra.run.smithi120.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-02-06T21:30:13.033 INFO:teuthology.orchestra.run.smithi120.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 630 kB/s | 32 kB 00:00 2024-02-06T21:30:13.057 INFO:teuthology.orchestra.run.smithi120.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 944 kB/s | 47 kB 00:00 2024-02-06T21:30:13.082 INFO:teuthology.orchestra.run.smithi120.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2024-02-06T21:30:13.107 INFO:teuthology.orchestra.run.smithi120.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-02-06T21:30:13.157 INFO:teuthology.orchestra.run.smithi120.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-02-06T21:30:13.183 INFO:teuthology.orchestra.run.smithi120.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-02-06T21:30:13.208 INFO:teuthology.orchestra.run.smithi120.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-02-06T21:30:13.233 INFO:teuthology.orchestra.run.smithi120.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-02-06T21:30:13.259 INFO:teuthology.orchestra.run.smithi120.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-02-06T21:30:13.336 INFO:teuthology.orchestra.run.smithi120.stdout:(17/36): ceph-mon-19.0.0-1269.g633ab857.el9.x86 3.1 MB/s | 4.7 MB 00:01 2024-02-06T21:30:13.403 INFO:teuthology.orchestra.run.smithi120.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 132 kB/s | 22 kB 00:00 2024-02-06T21:30:13.437 INFO:teuthology.orchestra.run.smithi120.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-02-06T21:30:13.488 INFO:teuthology.orchestra.run.smithi120.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 662 kB/s | 151 kB 00:00 2024-02-06T21:30:13.529 INFO:teuthology.orchestra.run.smithi120.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 258 kB/s | 11 kB 00:00 2024-02-06T21:30:13.571 INFO:teuthology.orchestra.run.smithi120.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-02-06T21:30:13.596 INFO:teuthology.orchestra.run.smithi120.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 265 kB/s | 18 kB 00:00 2024-02-06T21:30:13.621 INFO:teuthology.orchestra.run.smithi120.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-02-06T21:30:13.647 INFO:teuthology.orchestra.run.smithi120.stdout:(25/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 705 kB/s | 172 kB 00:00 2024-02-06T21:30:13.672 INFO:teuthology.orchestra.run.smithi120.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 256 kB/s | 19 kB 00:00 2024-02-06T21:30:13.697 INFO:teuthology.orchestra.run.smithi120.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-02-06T21:30:13.723 INFO:teuthology.orchestra.run.smithi120.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-02-06T21:30:13.748 INFO:teuthology.orchestra.run.smithi120.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-02-06T21:30:13.773 INFO:teuthology.orchestra.run.smithi120.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-02-06T21:30:13.798 INFO:teuthology.orchestra.run.smithi120.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-02-06T21:30:13.824 INFO:teuthology.orchestra.run.smithi120.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-02-06T21:30:13.849 INFO:teuthology.orchestra.run.smithi120.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-02-06T21:30:13.874 INFO:teuthology.orchestra.run.smithi120.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-02-06T21:30:13.899 INFO:teuthology.orchestra.run.smithi120.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-02-06T21:30:13.925 INFO:teuthology.orchestra.run.smithi120.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-02-06T21:30:13.928 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:13.928 INFO:teuthology.orchestra.run.smithi120.stdout:Total 12 MB/s | 30 MB 00:02 2024-02-06T21:30:13.949 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:30:13.989 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:30:13.989 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:30:14.413 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:30:14.413 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:30:15.771 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:30:15.836 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-06T21:30:15.883 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-06T21:30:15.922 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-06T21:30:15.993 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-06T21:30:16.030 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-06T21:30:16.088 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-06T21:30:16.137 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1269.g633ab857.el9.x86_64 5/6 2024-02-06T21:30:16.179 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-06T21:30:16.210 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-test-2:19.0.0-1269.g633ab857.el9.x86_64 6/6 2024-02-06T21:30:16.258 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-06T21:30:16.299 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-06T21:30:16.371 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-06T21:30:16.424 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-06T21:30:16.471 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-06T21:30:16.520 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-06T21:30:16.576 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-06T21:30:16.761 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-06T21:30:16.809 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-06T21:30:16.847 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-06T21:30:16.883 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-06T21:30:16.913 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-06T21:30:16.989 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-06T21:30:17.023 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-06T21:30:17.057 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-06T21:30:17.097 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-06T21:30:17.132 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-06T21:30:17.167 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-06T21:30:17.255 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-06T21:30:17.298 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-06T21:30:17.370 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-06T21:30:17.507 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-06T21:30:17.625 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-06T21:30:17.792 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1269.g633ab857.el9. 31/36 2024-02-06T21:30:17.821 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-2:19.0.0-1269.g633ab857.el9.x86_64 32/36 2024-02-06T21:30:17.846 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1269.g633ab857.el9.x86_64 32/36 2024-02-06T21:30:17.846 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:30:17.846 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-06T21:30:17.846 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-06T21:30:17.846 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-06T21:30:17.846 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:18.706 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-test-2:19.0.0-1269.g633ab857.el9.x86_64 6/6 2024-02-06T21:30:18.706 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-test-2:19.0.0-1269.g633ab857.el9.x86_64 1/6 2024-02-06T21:30:18.706 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephsqlite-2:19.0.0-1269.g633ab857.el9.x86_64 2/6 2024-02-06T21:30:18.706 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-06T21:30:18.706 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-06T21:30:18.706 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-06T21:30:19.138 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-06T21:30:19.138 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:19.138 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:30:19.138 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:19.138 INFO:teuthology.orchestra.run.smithi135.stdout: jq-1.6-16.el9.x86_64 2024-02-06T21:30:19.138 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:19.138 INFO:teuthology.orchestra.run.smithi135.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-06T21:30:19.138 INFO:teuthology.orchestra.run.smithi135.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-06T21:30:19.138 INFO:teuthology.orchestra.run.smithi135.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-06T21:30:19.138 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:19.139 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:30:19.288 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-osd-2:19.0.0-1269.g633ab857.el9.x86_64 33/36 2024-02-06T21:30:19.315 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-osd-2:19.0.0-1269.g633ab857.el9.x86_64 33/36 2024-02-06T21:30:19.315 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:30:19.315 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-06T21:30:19.315 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-06T21:30:19.315 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-06T21:30:19.315 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:19.436 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph 2024-02-06T21:30:19.764 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mon-2:19.0.0-1269.g633ab857.el9.x86_64 34/36 2024-02-06T21:30:19.787 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mon-2:19.0.0-1269.g633ab857.el9.x86_64 34/36 2024-02-06T21:30:19.787 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:30:19.787 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-06T21:30:19.787 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-06T21:30:19.787 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-06T21:30:19.787 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:19.998 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mds-2:19.0.0-1269.g633ab857.el9.x86_64 35/36 2024-02-06T21:30:20.007 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:41 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:30:20.021 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mds-2:19.0.0-1269.g633ab857.el9.x86_64 35/36 2024-02-06T21:30:20.021 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:30:20.021 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-06T21:30:20.021 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-06T21:30:20.021 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-06T21:30:20.022 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:20.088 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-2:19.0.0-1269.g633ab857.el9.x86_64 36/36 2024-02-06T21:30:20.168 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:30:20.170 INFO:teuthology.orchestra.run.smithi135.stdout:========================================================================================= 2024-02-06T21:30:20.171 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-02-06T21:30:20.171 INFO:teuthology.orchestra.run.smithi135.stdout:========================================================================================= 2024-02-06T21:30:20.171 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:30:20.171 INFO:teuthology.orchestra.run.smithi135.stdout: ceph x86_64 2:19.0.0-1269.g633ab857.el9 ceph 6.6 k 2024-02-06T21:30:20.171 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-02-06T21:30:20.171 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds x86_64 2:19.0.0-1269.g633ab857.el9 ceph 2.1 M 2024-02-06T21:30:20.171 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr x86_64 2:19.0.0-1269.g633ab857.el9 ceph 1.5 M 2024-02-06T21:30:20.171 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 247 k 2024-02-06T21:30:20.171 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon x86_64 2:19.0.0-1269.g633ab857.el9 ceph 4.7 M 2024-02-06T21:30:20.171 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd x86_64 2:19.0.0-1269.g633ab857.el9 ceph 17 M 2024-02-06T21:30:20.171 INFO:teuthology.orchestra.run.smithi135.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-06T21:30:20.171 INFO:teuthology.orchestra.run.smithi135.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-06T21:30:20.171 INFO:teuthology.orchestra.run.smithi135.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-06T21:30:20.171 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-06T21:30:20.172 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-06T21:30:20.172 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-06T21:30:20.172 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-06T21:30:20.172 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-06T21:30:20.172 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-06T21:30:20.172 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-06T21:30:20.172 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-06T21:30:20.172 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-06T21:30:20.172 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-06T21:30:20.172 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-06T21:30:20.172 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-06T21:30:20.172 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-06T21:30:20.172 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-06T21:30:20.172 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-06T21:30:20.173 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-06T21:30:20.173 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-06T21:30:20.173 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-06T21:30:20.173 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-06T21:30:20.173 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-06T21:30:20.173 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-06T21:30:20.173 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-06T21:30:20.173 INFO:teuthology.orchestra.run.smithi135.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-06T21:30:20.173 INFO:teuthology.orchestra.run.smithi135.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-06T21:30:20.173 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-06T21:30:20.173 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-06T21:30:20.173 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-06T21:30:20.173 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:20.173 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:30:20.174 INFO:teuthology.orchestra.run.smithi135.stdout:========================================================================================= 2024-02-06T21:30:20.174 INFO:teuthology.orchestra.run.smithi135.stdout:Install 36 Packages 2024-02-06T21:30:20.174 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:20.174 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 30 M 2024-02-06T21:30:20.174 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 107 M 2024-02-06T21:30:20.174 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:30:20.571 INFO:teuthology.orchestra.run.smithi135.stdout:(1/36): ceph-19.0.0-1269.g633ab857.el9.x86_64.r 30 kB/s | 6.6 kB 00:00 2024-02-06T21:30:20.729 INFO:teuthology.orchestra.run.smithi135.stdout:(2/36): ceph-mds-19.0.0-1269.g633ab857.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-02-06T21:30:20.988 INFO:teuthology.orchestra.run.smithi135.stdout:(3/36): ceph-mgr-19.0.0-1269.g633ab857.el9.x86_ 2.3 MB/s | 1.5 MB 00:00 2024-02-06T21:30:21.096 INFO:teuthology.orchestra.run.smithi135.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1269.g633a 2.2 MB/s | 247 kB 00:00 2024-02-06T21:30:21.205 INFO:teuthology.orchestra.run.smithi135.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-02-06T21:30:21.247 INFO:teuthology.orchestra.run.smithi135.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-02-06T21:30:21.272 INFO:teuthology.orchestra.run.smithi135.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-02-06T21:30:21.306 INFO:teuthology.orchestra.run.smithi135.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-02-06T21:30:21.339 INFO:teuthology.orchestra.run.smithi135.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-02-06T21:30:21.381 INFO:teuthology.orchestra.run.smithi135.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-02-06T21:30:21.565 INFO:teuthology.orchestra.run.smithi135.stdout:(11/36): ceph-osd-19.0.0-1269.g633ab857.el9.x86 20 MB/s | 17 MB 00:00 2024-02-06T21:30:21.607 INFO:teuthology.orchestra.run.smithi135.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 5.1 MB/s | 1.1 MB 00:00 2024-02-06T21:30:21.632 INFO:teuthology.orchestra.run.smithi135.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-02-06T21:30:21.657 INFO:teuthology.orchestra.run.smithi135.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-02-06T21:30:21.683 INFO:teuthology.orchestra.run.smithi135.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-02-06T21:30:21.708 INFO:teuthology.orchestra.run.smithi135.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-02-06T21:30:21.783 INFO:teuthology.orchestra.run.smithi135.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-02-06T21:30:21.834 INFO:teuthology.orchestra.run.smithi135.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-02-06T21:30:21.875 INFO:teuthology.orchestra.run.smithi135.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 473 kB/s | 43 kB 00:00 2024-02-06T21:30:21.901 INFO:teuthology.orchestra.run.smithi135.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-02-06T21:30:21.943 INFO:teuthology.orchestra.run.smithi135.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-02-06T21:30:21.963 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-2:19.0.0-1269.g633ab857.el9.x86_64 36/36 2024-02-06T21:30:21.963 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-2:19.0.0-1269.g633ab857.el9.x86_64 1/36 2024-02-06T21:30:21.963 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mds-2:19.0.0-1269.g633ab857.el9.x86_64 2/36 2024-02-06T21:30:21.963 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-2:19.0.0-1269.g633ab857.el9.x86_64 3/36 2024-02-06T21:30:21.963 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mon-2:19.0.0-1269.g633ab857.el9.x86_64 4/36 2024-02-06T21:30:21.963 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-osd-2:19.0.0-1269.g633ab857.el9.x86_64 5/36 2024-02-06T21:30:21.963 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1269.g633ab857.el9. 6/36 2024-02-06T21:30:21.964 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-06T21:30:21.964 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-06T21:30:21.964 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-06T21:30:21.964 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-06T21:30:21.964 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-06T21:30:21.964 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-06T21:30:21.964 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-06T21:30:21.964 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-06T21:30:21.964 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-06T21:30:21.964 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-06T21:30:21.964 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-06T21:30:21.964 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-06T21:30:21.964 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-06T21:30:21.965 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-06T21:30:21.965 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-06T21:30:21.965 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-06T21:30:21.965 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-06T21:30:21.965 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-06T21:30:21.965 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-06T21:30:21.965 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-06T21:30:21.965 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-06T21:30:21.965 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-06T21:30:21.965 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-06T21:30:21.965 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-06T21:30:21.965 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-06T21:30:21.965 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-06T21:30:21.965 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-06T21:30:21.965 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-06T21:30:21.966 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-06T21:30:21.968 INFO:teuthology.orchestra.run.smithi135.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-02-06T21:30:21.985 INFO:teuthology.orchestra.run.smithi135.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 423 kB/s | 18 kB 00:00 2024-02-06T21:30:22.003 INFO:teuthology.orchestra.run.smithi135.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 691 kB/s | 23 kB 00:00 2024-02-06T21:30:22.019 INFO:teuthology.orchestra.run.smithi135.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 580 kB/s | 19 kB 00:00 2024-02-06T21:30:22.036 INFO:teuthology.orchestra.run.smithi135.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 583 kB/s | 20 kB 00:00 2024-02-06T21:30:22.053 INFO:teuthology.orchestra.run.smithi135.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-02-06T21:30:22.078 INFO:teuthology.orchestra.run.smithi135.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-02-06T21:30:22.112 INFO:teuthology.orchestra.run.smithi135.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-02-06T21:30:22.129 INFO:teuthology.orchestra.run.smithi135.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-02-06T21:30:22.154 INFO:teuthology.orchestra.run.smithi135.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-02-06T21:30:22.171 INFO:teuthology.orchestra.run.smithi135.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 854 kB/s | 36 kB 00:00 2024-02-06T21:30:22.205 INFO:teuthology.orchestra.run.smithi135.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-02-06T21:30:22.230 INFO:teuthology.orchestra.run.smithi135.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.1 MB/s | 427 kB 00:00 2024-02-06T21:30:22.247 INFO:teuthology.orchestra.run.smithi135.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 477 kB/s | 20 kB 00:00 2024-02-06T21:30:22.431 INFO:teuthology.orchestra.run.smithi135.stdout:(36/36): ceph-mon-19.0.0-1269.g633ab857.el9.x86 2.5 MB/s | 4.7 MB 00:01 2024-02-06T21:30:22.434 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:22.434 INFO:teuthology.orchestra.run.smithi135.stdout:Total 13 MB/s | 30 MB 00:02 2024-02-06T21:30:22.454 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:30:22.498 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:30:22.499 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:30:22.908 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:30:22.909 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:30:23.091 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-06T21:30:23.091 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:23.091 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:30:23.091 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:23.092 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:23.092 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:23.092 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:30:23.092 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:23.092 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:23.092 INFO:teuthology.orchestra.run.smithi120.stdout: lua-5.4.4-4.el9.x86_64 2024-02-06T21:30:23.092 INFO:teuthology.orchestra.run.smithi120.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-06T21:30:23.092 INFO:teuthology.orchestra.run.smithi120.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-06T21:30:23.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-06T21:30:23.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-06T21:30:23.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-06T21:30:23.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-06T21:30:23.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-06T21:30:23.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-06T21:30:23.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-06T21:30:23.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-06T21:30:23.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-06T21:30:23.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-06T21:30:23.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-06T21:30:23.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-06T21:30:23.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-06T21:30:23.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-06T21:30:23.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-06T21:30:23.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply-3.11-14.el9.noarch 2024-02-06T21:30:23.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-06T21:30:23.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-06T21:30:23.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-06T21:30:23.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-06T21:30:23.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-06T21:30:23.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-06T21:30:23.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-06T21:30:23.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-06T21:30:23.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-06T21:30:23.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-06T21:30:23.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-06T21:30:23.094 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:23.094 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:30:23.438 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-base 2024-02-06T21:30:24.007 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:00 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:30:24.112 INFO:teuthology.orchestra.run.smithi120.stdout:Package ceph-base-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:30:24.163 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:30:24.164 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-02-06T21:30:24.164 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:30:24.227 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install cephadm 2024-02-06T21:30:24.286 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:30:24.333 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-06T21:30:24.366 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-06T21:30:24.394 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-06T21:30:24.465 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-06T21:30:24.517 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-06T21:30:24.550 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-06T21:30:24.644 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-06T21:30:24.712 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-06T21:30:24.754 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-06T21:30:24.789 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:00 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:30:24.826 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-06T21:30:24.880 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-06T21:30:24.918 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-06T21:30:24.945 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:30:24.946 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:30:24.946 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-02-06T21:30:24.946 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:30:24.946 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:30:24.946 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 748 k 2024-02-06T21:30:24.946 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:24.946 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:30:24.946 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:30:24.946 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:30:24.947 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:24.947 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 748 k 2024-02-06T21:30:24.947 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 754 k 2024-02-06T21:30:24.947 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:30:24.967 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-06T21:30:25.021 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-06T21:30:25.207 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-06T21:30:25.263 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-06T21:30:25.274 INFO:teuthology.orchestra.run.smithi120.stdout:cephadm-19.0.0-1269.g633ab857.el9.noarch.rpm 2.2 MB/s | 748 kB 00:00 2024-02-06T21:30:25.274 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:25.275 INFO:teuthology.orchestra.run.smithi120.stdout:Total 2.2 MB/s | 748 kB 00:00 2024-02-06T21:30:25.275 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:30:25.279 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:30:25.280 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:30:25.286 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:30:25.286 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:30:25.309 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-06T21:30:25.345 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-06T21:30:25.357 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:30:25.375 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-06T21:30:25.501 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-06T21:30:25.535 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-06T21:30:25.569 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-06T21:30:25.609 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-06T21:30:25.644 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-06T21:30:25.678 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-06T21:30:25.715 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:19.0.0-1269.g633ab857.el9.noarch 1/1 2024-02-06T21:30:25.767 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-06T21:30:25.795 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephadm-2:19.0.0-1269.g633ab857.el9.noarch 1/1 2024-02-06T21:30:25.817 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-06T21:30:25.890 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-06T21:30:26.027 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-06T21:30:26.153 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-06T21:30:26.184 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:19.0.0-1269.g633ab857.el9.noarch 1/1 2024-02-06T21:30:26.319 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1269.g633ab857.el9. 31/36 2024-02-06T21:30:26.347 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-2:19.0.0-1269.g633ab857.el9.x86_64 32/36 2024-02-06T21:30:26.371 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1269.g633ab857.el9.x86_64 32/36 2024-02-06T21:30:26.371 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:30:26.371 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-06T21:30:26.371 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-06T21:30:26.371 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-06T21:30:26.371 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:26.504 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephadm-2:19.0.0-1269.g633ab857.el9.noarch 1/1 2024-02-06T21:30:26.504 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:26.504 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:30:26.504 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:30:26.504 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:26.504 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:30:26.694 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-immutable-object-cache 2024-02-06T21:30:27.275 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:03 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:30:27.430 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:30:27.431 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:30:27.431 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:30:27.431 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:30:27.431 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:30:27.431 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1269.g633ab857.el9 ceph 145 k 2024-02-06T21:30:27.431 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:27.431 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:30:27.431 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:30:27.432 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:30:27.432 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:27.432 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 145 k 2024-02-06T21:30:27.432 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 436 k 2024-02-06T21:30:27.432 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:30:27.773 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-immutable-object-cache-19.0.0-1269.g633ab8 426 kB/s | 145 kB 00:00 2024-02-06T21:30:27.774 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:27.774 INFO:teuthology.orchestra.run.smithi120.stdout:Total 424 kB/s | 145 kB 00:00 2024-02-06T21:30:27.774 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:30:27.781 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:30:27.781 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:30:27.833 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:30:27.834 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:30:27.857 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-osd-2:19.0.0-1269.g633ab857.el9.x86_64 33/36 2024-02-06T21:30:27.885 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-osd-2:19.0.0-1269.g633ab857.el9.x86_64 33/36 2024-02-06T21:30:27.885 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:30:27.885 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-06T21:30:27.885 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-06T21:30:27.885 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-06T21:30:27.885 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:27.984 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:30:28.040 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1269.g633ab857. 1/1 2024-02-06T21:30:28.065 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1269.g633ab857. 1/1 2024-02-06T21:30:28.066 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:30:28.066 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-06T21:30:28.066 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:28.333 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mon-2:19.0.0-1269.g633ab857.el9.x86_64 34/36 2024-02-06T21:30:28.356 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mon-2:19.0.0-1269.g633ab857.el9.x86_64 34/36 2024-02-06T21:30:28.357 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:30:28.357 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-06T21:30:28.357 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-06T21:30:28.357 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-06T21:30:28.357 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:28.567 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mds-2:19.0.0-1269.g633ab857.el9.x86_64 35/36 2024-02-06T21:30:28.590 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mds-2:19.0.0-1269.g633ab857.el9.x86_64 35/36 2024-02-06T21:30:28.590 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:30:28.590 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-06T21:30:28.590 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-06T21:30:28.590 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-06T21:30:28.590 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:28.647 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-2:19.0.0-1269.g633ab857.el9.x86_64 36/36 2024-02-06T21:30:29.029 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1269.g633ab857. 1/1 2024-02-06T21:30:29.029 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:29.029 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:30:29.030 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:29.030 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:29.030 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:30:29.266 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr 2024-02-06T21:30:29.840 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:05 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:30:29.945 INFO:teuthology.orchestra.run.smithi120.stdout:Package ceph-mgr-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:30:29.997 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:30:29.998 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-02-06T21:30:29.999 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:30:30.065 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-dashboard 2024-02-06T21:30:30.631 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:06 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:30:30.648 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-2:19.0.0-1269.g633ab857.el9.x86_64 36/36 2024-02-06T21:30:30.648 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-2:19.0.0-1269.g633ab857.el9.x86_64 1/36 2024-02-06T21:30:30.648 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mds-2:19.0.0-1269.g633ab857.el9.x86_64 2/36 2024-02-06T21:30:30.648 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-2:19.0.0-1269.g633ab857.el9.x86_64 3/36 2024-02-06T21:30:30.648 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mon-2:19.0.0-1269.g633ab857.el9.x86_64 4/36 2024-02-06T21:30:30.648 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-osd-2:19.0.0-1269.g633ab857.el9.x86_64 5/36 2024-02-06T21:30:30.648 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1269.g633ab857.el9. 6/36 2024-02-06T21:30:30.648 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-06T21:30:30.649 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-06T21:30:30.649 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-06T21:30:30.649 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-06T21:30:30.649 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-06T21:30:30.649 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-06T21:30:30.649 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-06T21:30:30.649 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-06T21:30:30.649 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-06T21:30:30.649 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-06T21:30:30.649 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-06T21:30:30.649 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-06T21:30:30.649 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-06T21:30:30.649 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-06T21:30:30.649 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-06T21:30:30.650 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-06T21:30:30.650 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-06T21:30:30.650 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-06T21:30:30.650 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-06T21:30:30.651 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-06T21:30:30.651 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-06T21:30:30.651 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-06T21:30:30.651 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-06T21:30:30.651 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-06T21:30:30.651 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-06T21:30:30.651 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-06T21:30:30.651 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-06T21:30:30.651 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-06T21:30:30.652 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-06T21:30:30.787 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:30:30.788 INFO:teuthology.orchestra.run.smithi120.stdout:====================================================================================== 2024-02-06T21:30:30.788 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-02-06T21:30:30.789 INFO:teuthology.orchestra.run.smithi120.stdout:====================================================================================== 2024-02-06T21:30:30.789 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:30:30.789 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 1.8 M 2024-02-06T21:30:30.789 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-02-06T21:30:30.789 INFO:teuthology.orchestra.run.smithi120.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-06T21:30:30.789 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 28 k 2024-02-06T21:30:30.789 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 15 k 2024-02-06T21:30:30.789 INFO:teuthology.orchestra.run.smithi120.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-06T21:30:30.789 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-06T21:30:30.789 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-06T21:30:30.790 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-06T21:30:30.790 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-06T21:30:30.790 INFO:teuthology.orchestra.run.smithi120.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-06T21:30:30.790 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-06T21:30:30.790 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-06T21:30:30.790 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:30.790 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:30:30.790 INFO:teuthology.orchestra.run.smithi120.stdout:====================================================================================== 2024-02-06T21:30:30.790 INFO:teuthology.orchestra.run.smithi120.stdout:Install 12 Packages 2024-02-06T21:30:30.790 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:30.791 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 6.8 M 2024-02-06T21:30:30.791 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 26 M 2024-02-06T21:30:30.791 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:30:31.236 INFO:teuthology.orchestra.run.smithi120.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1269.g63 149 kB/s | 28 kB 00:00 2024-02-06T21:30:31.336 INFO:teuthology.orchestra.run.smithi120.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1269.g633 53 kB/s | 15 kB 00:00 2024-02-06T21:30:31.478 INFO:teuthology.orchestra.run.smithi120.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2024-02-06T21:30:31.504 INFO:teuthology.orchestra.run.smithi120.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 238 kB 00:00 2024-02-06T21:30:31.619 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-06T21:30:31.619 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:31.619 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:30:31.619 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:31.619 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:31.619 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:31.619 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:30:31.619 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:31.619 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:31.619 INFO:teuthology.orchestra.run.smithi135.stdout: lua-5.4.4-4.el9.x86_64 2024-02-06T21:30:31.619 INFO:teuthology.orchestra.run.smithi135.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-06T21:30:31.620 INFO:teuthology.orchestra.run.smithi135.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-06T21:30:31.620 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-06T21:30:31.620 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-06T21:30:31.620 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-06T21:30:31.620 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-06T21:30:31.620 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-06T21:30:31.620 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-06T21:30:31.620 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-06T21:30:31.620 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-06T21:30:31.620 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-06T21:30:31.620 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-06T21:30:31.620 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-06T21:30:31.620 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-06T21:30:31.620 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-06T21:30:31.621 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-06T21:30:31.621 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-06T21:30:31.621 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ply-3.11-14.el9.noarch 2024-02-06T21:30:31.621 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-06T21:30:31.621 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-06T21:30:31.621 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-06T21:30:31.621 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-06T21:30:31.621 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-06T21:30:31.621 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-06T21:30:31.621 INFO:teuthology.orchestra.run.smithi135.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-06T21:30:31.621 INFO:teuthology.orchestra.run.smithi135.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-06T21:30:31.621 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-06T21:30:31.621 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-06T21:30:31.621 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-06T21:30:31.622 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:31.622 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:30:31.729 INFO:teuthology.orchestra.run.smithi120.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1269.g633ab85 2.6 MB/s | 1.8 MB 00:00 2024-02-06T21:30:31.979 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-base 2024-02-06T21:30:31.987 INFO:teuthology.orchestra.run.smithi120.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-02-06T21:30:32.062 INFO:teuthology.orchestra.run.smithi120.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 982 kB/s | 548 kB 00:00 2024-02-06T21:30:32.204 INFO:teuthology.orchestra.run.smithi120.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.0 MB/s | 144 kB 00:00 2024-02-06T21:30:32.320 INFO:teuthology.orchestra.run.smithi120.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 264 kB/s | 31 kB 00:00 2024-02-06T21:30:32.546 INFO:teuthology.orchestra.run.smithi120.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 836 kB/s | 188 kB 00:00 2024-02-06T21:30:32.560 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:53 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:30:32.588 INFO:teuthology.orchestra.run.smithi120.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.4 MB/s | 2.0 MB 00:00 2024-02-06T21:30:32.664 INFO:teuthology.orchestra.run.smithi135.stdout:Package ceph-base-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:30:32.716 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:30:32.717 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-02-06T21:30:32.717 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:30:32.784 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install cephadm 2024-02-06T21:30:32.922 INFO:teuthology.orchestra.run.smithi120.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 598 kB/s | 863 kB 00:01 2024-02-06T21:30:32.925 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:32.925 INFO:teuthology.orchestra.run.smithi120.stdout:Total 3.2 MB/s | 6.8 MB 00:02 2024-02-06T21:30:32.945 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:30:32.964 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:30:32.964 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:30:33.149 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:30:33.150 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:30:33.353 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:54 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:30:33.511 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:30:33.512 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:30:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-02-06T21:30:33.512 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:30:33.512 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:30:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 748 k 2024-02-06T21:30:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:33.512 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:30:33.513 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:30:33.513 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:30:33.513 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:33.513 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 748 k 2024-02-06T21:30:33.513 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 754 k 2024-02-06T21:30:33.513 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:30:33.655 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:30:33.730 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-02-06T21:30:33.774 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-02-06T21:30:33.820 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-02-06T21:30:33.840 INFO:teuthology.orchestra.run.smithi135.stdout:cephadm-19.0.0-1269.g633ab857.el9.noarch.rpm 2.2 MB/s | 748 kB 00:00 2024-02-06T21:30:33.841 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:33.841 INFO:teuthology.orchestra.run.smithi135.stdout:Total 2.2 MB/s | 748 kB 00:00 2024-02-06T21:30:33.841 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:30:33.846 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:30:33.846 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:30:33.852 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:30:33.852 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:30:33.859 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-02-06T21:30:33.928 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:30:33.965 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-02-06T21:30:34.096 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-02-06T21:30:34.179 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-02-06T21:30:34.223 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-02-06T21:30:34.252 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-02-06T21:30:34.282 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1269.g633ab857.el9 10/12 2024-02-06T21:30:34.332 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephadm-2:19.0.0-1269.g633ab857.el9.noarch 1/1 2024-02-06T21:30:34.411 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cephadm-2:19.0.0-1269.g633ab857.el9.noarch 1/1 2024-02-06T21:30:34.525 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1269.g633ab857.el 11/12 2024-02-06T21:30:34.559 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1269.g633ab857.el9.noa 12/12 2024-02-06T21:30:34.847 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephadm-2:19.0.0-1269.g633ab857.el9.noarch 1/1 2024-02-06T21:30:35.217 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cephadm-2:19.0.0-1269.g633ab857.el9.noarch 1/1 2024-02-06T21:30:35.217 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:35.217 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:30:35.217 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:30:35.218 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:35.218 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:30:35.410 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-immutable-object-cache 2024-02-06T21:30:35.412 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1269.g633ab857.el9.noa 12/12 2024-02-06T21:30:35.412 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1269.g633ab857.el 1/12 2024-02-06T21:30:35.412 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1269.g633ab857.el9.noa 2/12 2024-02-06T21:30:35.413 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1269.g633ab857.el9 3/12 2024-02-06T21:30:35.413 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-02-06T21:30:35.413 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-02-06T21:30:35.413 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-02-06T21:30:35.413 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-02-06T21:30:35.413 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-02-06T21:30:35.413 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-02-06T21:30:35.413 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-02-06T21:30:35.413 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-02-06T21:30:35.775 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-02-06T21:30:35.776 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:35.776 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:30:35.776 INFO:teuthology.orchestra.run.smithi120.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-06T21:30:35.776 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:30:35.776 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:30:35.776 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:30:35.776 INFO:teuthology.orchestra.run.smithi120.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-06T21:30:35.776 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-06T21:30:35.776 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-06T21:30:35.777 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-06T21:30:35.777 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-06T21:30:35.777 INFO:teuthology.orchestra.run.smithi120.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-06T21:30:35.777 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-06T21:30:35.777 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-06T21:30:35.777 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:35.778 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:30:35.986 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:56 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:30:36.029 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-06T21:30:36.147 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:30:36.148 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:30:36.148 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:30:36.148 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:30:36.148 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:30:36.148 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1269.g633ab857.el9 ceph 145 k 2024-02-06T21:30:36.148 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:36.148 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:30:36.149 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:30:36.149 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:30:36.149 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:36.149 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 145 k 2024-02-06T21:30:36.149 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 436 k 2024-02-06T21:30:36.149 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:30:36.496 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-immutable-object-cache-19.0.0-1269.g633ab8 418 kB/s | 145 kB 00:00 2024-02-06T21:30:36.497 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:36.497 INFO:teuthology.orchestra.run.smithi135.stdout:Total 417 kB/s | 145 kB 00:00 2024-02-06T21:30:36.497 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:30:36.505 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:30:36.505 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:30:36.556 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:30:36.557 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:30:36.597 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:12 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:30:36.704 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:30:36.755 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:30:36.761 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2024-02-06T21:30:36.761 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-02-06T21:30:36.761 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2024-02-06T21:30:36.761 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:30:36.761 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 7.4 M 2024-02-06T21:30:36.761 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-02-06T21:30:36.761 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-06T21:30:36.762 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-06T21:30:36.762 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-06T21:30:36.762 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-06T21:30:36.762 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-06T21:30:36.762 INFO:teuthology.orchestra.run.smithi120.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-06T21:30:36.762 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-06T21:30:36.762 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-06T21:30:36.763 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-06T21:30:36.763 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-06T21:30:36.763 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-06T21:30:36.763 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:36.763 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:30:36.763 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2024-02-06T21:30:36.763 INFO:teuthology.orchestra.run.smithi120.stdout:Install 12 Packages 2024-02-06T21:30:36.764 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:36.764 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 37 M 2024-02-06T21:30:36.764 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 209 M 2024-02-06T21:30:36.764 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:30:36.765 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1269.g633ab857. 1/1 2024-02-06T21:30:36.789 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1269.g633ab857. 1/1 2024-02-06T21:30:36.789 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:30:36.789 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-06T21:30:36.789 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:36.947 INFO:teuthology.orchestra.run.smithi120.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 216 kB/s | 32 kB 00:00 2024-02-06T21:30:36.980 INFO:teuthology.orchestra.run.smithi120.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 523 kB/s | 17 kB 00:00 2024-02-06T21:30:37.047 INFO:teuthology.orchestra.run.smithi120.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-02-06T21:30:37.073 INFO:teuthology.orchestra.run.smithi120.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 377 kB/s | 34 kB 00:00 2024-02-06T21:30:37.256 INFO:teuthology.orchestra.run.smithi120.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-02-06T21:30:37.325 INFO:teuthology.orchestra.run.smithi120.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-02-06T21:30:37.348 INFO:teuthology.orchestra.run.smithi120.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-02-06T21:30:37.490 INFO:teuthology.orchestra.run.smithi120.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.6 MB/s | 802 kB 00:00 2024-02-06T21:30:37.691 INFO:teuthology.orchestra.run.smithi120.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-02-06T21:30:37.716 INFO:teuthology.orchestra.run.smithi120.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 870 kB/s | 194 kB 00:00 2024-02-06T21:30:37.875 INFO:teuthology.orchestra.run.smithi120.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 6.9 MB/s | 7.4 MB 00:01 2024-02-06T21:30:37.882 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1269.g633ab857. 1/1 2024-02-06T21:30:37.882 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:37.882 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:30:37.882 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:30:37.882 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:37.882 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:30:38.025 INFO:teuthology.orchestra.run.smithi120.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 736 kB/s | 245 kB 00:00 2024-02-06T21:30:38.028 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:38.028 INFO:teuthology.orchestra.run.smithi120.stdout:Total 29 MB/s | 37 MB 00:01 2024-02-06T21:30:38.041 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:30:38.065 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:30:38.066 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:30:38.171 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr 2024-02-06T21:30:38.426 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:30:38.427 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:30:38.756 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:59 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:30:38.863 INFO:teuthology.orchestra.run.smithi135.stdout:Package ceph-mgr-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:30:38.915 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:30:38.916 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-02-06T21:30:38.916 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:30:38.983 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-dashboard 2024-02-06T21:30:39.129 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:30:39.197 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-06T21:30:39.229 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-06T21:30:39.322 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-06T21:30:39.357 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-06T21:30:39.527 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-06T21:30:39.551 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:00 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:30:39.556 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-06T21:30:39.632 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-06T21:30:39.709 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:30:39.710 INFO:teuthology.orchestra.run.smithi135.stdout:====================================================================================== 2024-02-06T21:30:39.710 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-02-06T21:30:39.710 INFO:teuthology.orchestra.run.smithi135.stdout:====================================================================================== 2024-02-06T21:30:39.710 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:30:39.710 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 1.8 M 2024-02-06T21:30:39.710 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-02-06T21:30:39.710 INFO:teuthology.orchestra.run.smithi135.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-06T21:30:39.710 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 28 k 2024-02-06T21:30:39.710 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 15 k 2024-02-06T21:30:39.710 INFO:teuthology.orchestra.run.smithi135.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-06T21:30:39.710 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-06T21:30:39.710 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-06T21:30:39.711 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-06T21:30:39.711 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-06T21:30:39.711 INFO:teuthology.orchestra.run.smithi135.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-06T21:30:39.711 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-06T21:30:39.711 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-06T21:30:39.711 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:39.711 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:30:39.711 INFO:teuthology.orchestra.run.smithi135.stdout:====================================================================================== 2024-02-06T21:30:39.711 INFO:teuthology.orchestra.run.smithi135.stdout:Install 12 Packages 2024-02-06T21:30:39.711 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:39.712 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 6.8 M 2024-02-06T21:30:39.712 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 26 M 2024-02-06T21:30:39.712 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:30:40.192 INFO:teuthology.orchestra.run.smithi135.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1269.g63 132 kB/s | 28 kB 00:00 2024-02-06T21:30:40.217 INFO:teuthology.orchestra.run.smithi135.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1269.g633 64 kB/s | 15 kB 00:00 2024-02-06T21:30:40.288 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-06T21:30:40.342 INFO:teuthology.orchestra.run.smithi135.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1269.g633ab85 4.8 MB/s | 1.8 MB 00:00 2024-02-06T21:30:40.402 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-06T21:30:40.509 INFO:teuthology.orchestra.run.smithi135.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 889 kB/s | 238 kB 00:00 2024-02-06T21:30:40.593 INFO:teuthology.orchestra.run.smithi135.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.5 MB/s | 1.0 MB 00:00 2024-02-06T21:30:40.643 INFO:teuthology.orchestra.run.smithi135.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.0 MB/s | 548 kB 00:00 2024-02-06T21:30:40.668 INFO:teuthology.orchestra.run.smithi135.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 265 kB/s | 19 kB 00:00 2024-02-06T21:30:40.735 INFO:teuthology.orchestra.run.smithi135.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 22 MB/s | 2.0 MB 00:00 2024-02-06T21:30:40.761 INFO:teuthology.orchestra.run.smithi135.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.5 MB/s | 144 kB 00:00 2024-02-06T21:30:40.786 INFO:teuthology.orchestra.run.smithi135.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 610 kB/s | 31 kB 00:00 2024-02-06T21:30:40.811 INFO:teuthology.orchestra.run.smithi135.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-02-06T21:30:41.220 INFO:teuthology.orchestra.run.smithi135.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 986 kB/s | 863 kB 00:00 2024-02-06T21:30:41.224 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:41.224 INFO:teuthology.orchestra.run.smithi135.stdout:Total 4.5 MB/s | 6.8 MB 00:01 2024-02-06T21:30:41.242 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:30:41.259 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:30:41.259 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:30:41.427 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:30:41.428 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:30:41.621 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-06T21:30:41.954 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:30:42.028 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-02-06T21:30:42.072 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-02-06T21:30:42.118 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-02-06T21:30:42.157 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-02-06T21:30:42.254 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-02-06T21:30:42.381 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-02-06T21:30:42.484 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-02-06T21:30:42.528 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-02-06T21:30:42.557 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-02-06T21:30:42.587 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1269.g633ab857.el9 10/12 2024-02-06T21:30:42.657 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-06T21:30:42.691 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1269.g633ab 12/12 2024-02-06T21:30:42.830 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1269.g633ab857.el 11/12 2024-02-06T21:30:42.864 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1269.g633ab857.el9.noa 12/12 2024-02-06T21:30:43.692 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1269.g633ab857.el9.noa 12/12 2024-02-06T21:30:43.692 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1269.g633ab857.el 1/12 2024-02-06T21:30:43.693 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1269.g633ab857.el9.noa 2/12 2024-02-06T21:30:43.693 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1269.g633ab857.el9 3/12 2024-02-06T21:30:43.693 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-02-06T21:30:43.693 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-02-06T21:30:43.693 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-02-06T21:30:43.693 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-02-06T21:30:43.693 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-02-06T21:30:43.693 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-02-06T21:30:43.693 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-02-06T21:30:43.693 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-02-06T21:30:44.051 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-02-06T21:30:44.051 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:44.052 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:30:44.052 INFO:teuthology.orchestra.run.smithi135.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-06T21:30:44.052 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:30:44.052 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:30:44.052 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:30:44.052 INFO:teuthology.orchestra.run.smithi135.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-06T21:30:44.052 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-06T21:30:44.052 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-06T21:30:44.052 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-06T21:30:44.052 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-06T21:30:44.052 INFO:teuthology.orchestra.run.smithi135.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-06T21:30:44.052 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-06T21:30:44.052 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-06T21:30:44.052 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:44.053 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:30:44.320 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-06T21:30:44.892 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:05 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:30:45.051 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:30:45.052 INFO:teuthology.orchestra.run.smithi135.stdout:============================================================================================ 2024-02-06T21:30:45.052 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-02-06T21:30:45.052 INFO:teuthology.orchestra.run.smithi135.stdout:============================================================================================ 2024-02-06T21:30:45.053 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:30:45.053 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 7.4 M 2024-02-06T21:30:45.053 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-02-06T21:30:45.053 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-06T21:30:45.053 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-06T21:30:45.053 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-06T21:30:45.053 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-06T21:30:45.053 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-06T21:30:45.053 INFO:teuthology.orchestra.run.smithi135.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-06T21:30:45.053 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-06T21:30:45.053 INFO:teuthology.orchestra.run.smithi135.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-06T21:30:45.053 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-06T21:30:45.053 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-06T21:30:45.054 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-06T21:30:45.054 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:45.054 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:30:45.054 INFO:teuthology.orchestra.run.smithi135.stdout:============================================================================================ 2024-02-06T21:30:45.054 INFO:teuthology.orchestra.run.smithi135.stdout:Install 12 Packages 2024-02-06T21:30:45.054 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:45.054 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 37 M 2024-02-06T21:30:45.055 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 209 M 2024-02-06T21:30:45.055 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:30:45.234 INFO:teuthology.orchestra.run.smithi135.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 229 kB/s | 32 kB 00:00 2024-02-06T21:30:45.284 INFO:teuthology.orchestra.run.smithi135.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 345 kB/s | 17 kB 00:00 2024-02-06T21:30:45.335 INFO:teuthology.orchestra.run.smithi135.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 704 kB/s | 34 kB 00:00 2024-02-06T21:30:45.355 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1269.g633ab 12/12 2024-02-06T21:30:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1269.g633ab 1/12 2024-02-06T21:30:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-06T21:30:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-06T21:30:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-06T21:30:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-06T21:30:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-06T21:30:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-06T21:30:45.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-06T21:30:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-06T21:30:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-06T21:30:45.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-06T21:30:45.385 INFO:teuthology.orchestra.run.smithi135.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-02-06T21:30:45.669 INFO:teuthology.orchestra.run.smithi135.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-02-06T21:30:45.736 INFO:teuthology.orchestra.run.smithi135.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 4.8 MB 00:00 2024-02-06T21:30:45.761 INFO:teuthology.orchestra.run.smithi135.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-02-06T21:30:45.832 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-06T21:30:45.833 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:45.833 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:30:45.833 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:30:45.833 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-06T21:30:45.833 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-06T21:30:45.833 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-06T21:30:45.833 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-06T21:30:45.833 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-06T21:30:45.833 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-06T21:30:45.833 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-06T21:30:45.833 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-06T21:30:45.833 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-06T21:30:45.833 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-06T21:30:45.834 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-06T21:30:45.834 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:45.834 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:30:45.861 INFO:teuthology.orchestra.run.smithi135.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-12 9.7 MB/s | 7.4 MB 00:00 2024-02-06T21:30:46.045 INFO:teuthology.orchestra.run.smithi135.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 802 kB 00:00 2024-02-06T21:30:46.112 INFO:teuthology.orchestra.run.smithi135.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 784 kB/s | 194 kB 00:00 2024-02-06T21:30:46.229 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-rook 2024-02-06T21:30:46.262 INFO:teuthology.orchestra.run.smithi135.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-02-06T21:30:46.337 INFO:teuthology.orchestra.run.smithi135.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 844 kB/s | 245 kB 00:00 2024-02-06T21:30:46.339 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:46.340 INFO:teuthology.orchestra.run.smithi135.stdout:Total 29 MB/s | 37 MB 00:01 2024-02-06T21:30:46.352 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:30:46.379 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:30:46.380 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:30:46.743 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:30:46.743 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:30:46.806 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:22 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:30:46.960 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:30:46.961 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-02-06T21:30:46.962 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-02-06T21:30:46.962 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-02-06T21:30:46.962 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:30:46.962 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 52 k 2024-02-06T21:30:46.962 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-02-06T21:30:46.962 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-06T21:30:46.962 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-06T21:30:46.962 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-06T21:30:46.962 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-06T21:30:46.962 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-06T21:30:46.962 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-06T21:30:46.962 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-06T21:30:46.962 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-06T21:30:46.963 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-06T21:30:46.963 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-06T21:30:46.963 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-06T21:30:46.963 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-06T21:30:46.963 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:46.963 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:30:46.963 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-02-06T21:30:46.963 INFO:teuthology.orchestra.run.smithi120.stdout:Install 13 Packages 2024-02-06T21:30:46.963 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:46.964 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 2.0 M 2024-02-06T21:30:46.964 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 26 M 2024-02-06T21:30:46.964 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:30:47.444 INFO:teuthology.orchestra.run.smithi120.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 88 kB/s | 18 kB 00:00 2024-02-06T21:30:47.469 INFO:teuthology.orchestra.run.smithi120.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 106 kB/s | 24 kB 00:00 2024-02-06T21:30:47.477 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:30:47.494 INFO:teuthology.orchestra.run.smithi120.stdout:(3/13): ceph-mgr-rook-19.0.0-1269.g633ab857.el9 205 kB/s | 52 kB 00:00 2024-02-06T21:30:47.551 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-06T21:30:47.583 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-06T21:30:47.612 INFO:teuthology.orchestra.run.smithi120.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.0 MB/s | 174 kB 00:00 2024-02-06T21:30:47.637 INFO:teuthology.orchestra.run.smithi120.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 790 kB/s | 132 kB 00:00 2024-02-06T21:30:47.670 INFO:teuthology.orchestra.run.smithi120.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 211 kB 00:00 2024-02-06T21:30:47.685 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-06T21:30:47.696 INFO:teuthology.orchestra.run.smithi120.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 510 kB/s | 43 kB 00:00 2024-02-06T21:30:47.728 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-06T21:30:47.905 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-06T21:30:47.935 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-06T21:30:47.962 INFO:teuthology.orchestra.run.smithi120.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-02-06T21:30:47.988 INFO:teuthology.orchestra.run.smithi120.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 92 kB/s | 32 kB 00:00 2024-02-06T21:30:48.012 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-06T21:30:48.188 INFO:teuthology.orchestra.run.smithi120.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 296 kB/s | 59 kB 00:00 2024-02-06T21:30:48.230 INFO:teuthology.orchestra.run.smithi120.stdout:(11/13): python3-google-auth-2.27.0-1.el9.noarc 379 kB/s | 203 kB 00:00 2024-02-06T21:30:48.397 INFO:teuthology.orchestra.run.smithi120.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 430 kB/s | 90 kB 00:00 2024-02-06T21:30:48.647 INFO:teuthology.orchestra.run.smithi120.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2024-02-06T21:30:48.649 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:48.650 INFO:teuthology.orchestra.run.smithi120.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-02-06T21:30:48.650 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:30:48.665 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:30:48.665 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:30:48.681 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-06T21:30:48.772 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:30:48.774 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:30:48.798 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-06T21:30:49.333 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:30:49.512 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-06T21:30:49.573 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-06T21:30:49.629 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-06T21:30:49.666 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-06T21:30:49.708 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-06T21:30:49.803 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-06T21:30:49.920 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-06T21:30:49.976 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-06T21:30:50.032 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-06T21:30:50.704 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-06T21:30:50.757 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-06T21:30:50.792 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-06T21:30:50.839 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-06T21:30:50.853 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-rook-2:19.0.0-1269.g633ab857.el9.noarch 13/13 2024-02-06T21:30:51.066 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-06T21:30:51.093 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1269.g633ab 12/12 2024-02-06T21:30:51.661 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1269.g633ab857.el9.noarch 13/13 2024-02-06T21:30:51.661 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1269.g633ab857.el9.noarch 1/13 2024-02-06T21:30:51.661 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-06T21:30:51.661 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-06T21:30:51.661 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-06T21:30:51.661 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-06T21:30:51.661 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-06T21:30:51.661 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-06T21:30:51.661 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-06T21:30:51.661 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-06T21:30:51.662 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-06T21:30:51.662 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-06T21:30:51.662 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-06T21:30:52.054 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-06T21:30:52.055 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:52.055 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:30:52.055 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:30:52.055 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-06T21:30:52.055 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-06T21:30:52.055 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-06T21:30:52.055 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-06T21:30:52.055 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-06T21:30:52.055 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-06T21:30:52.055 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-06T21:30:52.055 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-06T21:30:52.055 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-06T21:30:52.055 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-06T21:30:52.055 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-06T21:30:52.056 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-06T21:30:52.056 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:52.056 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:30:52.304 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-cephadm 2024-02-06T21:30:52.874 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:28 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:30:53.035 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:30:53.036 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-02-06T21:30:53.036 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-02-06T21:30:53.036 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-02-06T21:30:53.036 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:30:53.036 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 138 k 2024-02-06T21:30:53.036 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-02-06T21:30:53.036 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-06T21:30:53.037 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-06T21:30:53.037 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-06T21:30:53.037 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-06T21:30:53.037 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-06T21:30:53.037 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:53.037 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:30:53.037 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-02-06T21:30:53.037 INFO:teuthology.orchestra.run.smithi120.stdout:Install 6 Packages 2024-02-06T21:30:53.037 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:53.038 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 6.8 M 2024-02-06T21:30:53.038 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 33 M 2024-02-06T21:30:53.038 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:30:53.360 INFO:teuthology.orchestra.run.smithi120.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 227 kB 00:00 2024-02-06T21:30:53.543 INFO:teuthology.orchestra.run.smithi120.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-02-06T21:30:53.627 INFO:teuthology.orchestra.run.smithi120.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1269.g633ab857.e 314 kB/s | 138 kB 00:00 2024-02-06T21:30:53.702 INFO:teuthology.orchestra.run.smithi120.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 364 kB/s | 58 kB 00:00 2024-02-06T21:30:53.752 INFO:teuthology.orchestra.run.smithi120.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2024-02-06T21:30:53.781 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1269.g633ab 12/12 2024-02-06T21:30:53.781 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1269.g633ab 1/12 2024-02-06T21:30:53.781 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-06T21:30:53.781 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-06T21:30:53.781 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-06T21:30:53.781 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-06T21:30:53.781 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-06T21:30:53.781 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-06T21:30:53.782 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-06T21:30:53.782 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-06T21:30:53.782 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-06T21:30:53.782 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-06T21:30:53.827 INFO:teuthology.orchestra.run.smithi120.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 255 kB/s | 51 kB 00:00 2024-02-06T21:30:53.831 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:53.831 INFO:teuthology.orchestra.run.smithi120.stdout:Total 8.5 MB/s | 6.8 MB 00:00 2024-02-06T21:30:53.831 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:30:53.840 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:30:53.840 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:30:53.931 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:30:53.931 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:30:54.238 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:30:54.253 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-06T21:30:54.253 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:54.253 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:30:54.254 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:30:54.254 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-06T21:30:54.254 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-06T21:30:54.254 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-06T21:30:54.254 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-06T21:30:54.254 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-06T21:30:54.254 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-06T21:30:54.254 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-06T21:30:54.254 INFO:teuthology.orchestra.run.smithi135.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-06T21:30:54.254 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-06T21:30:54.254 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-06T21:30:54.254 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-06T21:30:54.254 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:54.254 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:30:54.398 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-06T21:30:54.462 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-06T21:30:54.626 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-rook 2024-02-06T21:30:54.975 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-06T21:30:55.057 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-06T21:30:55.119 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-06T21:30:55.142 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1269.g633ab857.el9.noarch 6/6 2024-02-06T21:30:55.206 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:16 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:30:55.363 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:30:55.364 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2024-02-06T21:30:55.365 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-02-06T21:30:55.365 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2024-02-06T21:30:55.365 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:30:55.365 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 52 k 2024-02-06T21:30:55.365 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-02-06T21:30:55.365 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-06T21:30:55.365 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-06T21:30:55.365 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-06T21:30:55.365 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-06T21:30:55.365 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-06T21:30:55.365 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-06T21:30:55.365 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-06T21:30:55.366 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-06T21:30:55.366 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-06T21:30:55.366 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-06T21:30:55.366 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-06T21:30:55.366 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-06T21:30:55.366 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:55.366 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:30:55.366 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2024-02-06T21:30:55.366 INFO:teuthology.orchestra.run.smithi135.stdout:Install 13 Packages 2024-02-06T21:30:55.366 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:55.367 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 2.0 M 2024-02-06T21:30:55.367 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 26 M 2024-02-06T21:30:55.367 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:30:55.642 INFO:teuthology.orchestra.run.smithi135.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 142 kB/s | 18 kB 00:00 2024-02-06T21:30:55.668 INFO:teuthology.orchestra.run.smithi135.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 160 kB/s | 24 kB 00:00 2024-02-06T21:30:55.718 INFO:teuthology.orchestra.run.smithi135.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-02-06T21:30:55.735 INFO:teuthology.orchestra.run.smithi135.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-02-06T21:30:55.760 INFO:teuthology.orchestra.run.smithi135.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-02-06T21:30:55.785 INFO:teuthology.orchestra.run.smithi135.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 849 kB/s | 43 kB 00:00 2024-02-06T21:30:55.811 INFO:teuthology.orchestra.run.smithi135.stdout:(7/13): ceph-mgr-rook-19.0.0-1269.g633ab857.el9 176 kB/s | 52 kB 00:00 2024-02-06T21:30:56.006 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1269.g633ab857.el9.noarch 6/6 2024-02-06T21:30:56.006 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1269.g633ab857.el9.noarch 1/6 2024-02-06T21:30:56.006 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-06T21:30:56.007 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-06T21:30:56.007 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-06T21:30:56.007 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-06T21:30:56.053 INFO:teuthology.orchestra.run.smithi135.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-02-06T21:30:56.078 INFO:teuthology.orchestra.run.smithi135.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 101 kB/s | 32 kB 00:00 2024-02-06T21:30:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-06T21:30:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:56.305 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:30:56.305 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:30:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-06T21:30:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-06T21:30:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-06T21:30:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-06T21:30:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-06T21:30:56.306 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:56.306 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:30:56.312 INFO:teuthology.orchestra.run.smithi135.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 405 kB/s | 203 kB 00:00 2024-02-06T21:30:56.337 INFO:teuthology.orchestra.run.smithi135.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 228 kB/s | 59 kB 00:00 2024-02-06T21:30:56.420 INFO:teuthology.orchestra.run.smithi135.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 826 kB/s | 90 kB 00:00 2024-02-06T21:30:56.569 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-fuse 2024-02-06T21:30:56.587 INFO:teuthology.orchestra.run.smithi135.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-02-06T21:30:56.589 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:56.589 INFO:teuthology.orchestra.run.smithi135.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-02-06T21:30:56.590 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:30:56.604 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:30:56.604 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:30:56.708 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:30:56.708 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:30:57.131 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:33 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:30:57.227 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:30:57.291 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:30:57.292 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:30:57.292 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-02-06T21:30:57.292 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:30:57.292 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:30:57.293 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse x86_64 2:19.0.0-1269.g633ab857.el9 ceph 886 k 2024-02-06T21:30:57.293 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-02-06T21:30:57.293 INFO:teuthology.orchestra.run.smithi120.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-06T21:30:57.293 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:57.293 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:30:57.293 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:30:57.293 INFO:teuthology.orchestra.run.smithi120.stdout:Install 2 Packages 2024-02-06T21:30:57.293 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:57.293 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 964 k 2024-02-06T21:30:57.293 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 2.8 M 2024-02-06T21:30:57.293 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:30:57.410 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-06T21:30:57.421 INFO:teuthology.orchestra.run.smithi120.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 614 kB/s | 79 kB 00:00 2024-02-06T21:30:57.471 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-06T21:30:57.526 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-06T21:30:57.556 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-06T21:30:57.597 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-06T21:30:57.605 INFO:teuthology.orchestra.run.smithi120.stdout:(2/2): ceph-fuse-19.0.0-1269.g633ab857.el9.x86_ 2.8 MB/s | 886 kB 00:00 2024-02-06T21:30:57.606 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:30:57.607 INFO:teuthology.orchestra.run.smithi120.stdout:Total 3.0 MB/s | 964 kB 00:00 2024-02-06T21:30:57.607 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:30:57.627 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:30:57.627 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:30:57.691 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-06T21:30:57.702 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:30:57.703 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:30:57.809 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-06T21:30:57.857 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-06T21:30:57.886 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:30:58.032 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-06T21:30:58.113 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 2/2 2024-02-06T21:30:58.580 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-06T21:30:58.637 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-06T21:30:58.671 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-06T21:30:58.728 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-06T21:30:58.749 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-rook-2:19.0.0-1269.g633ab857.el9.noarch 13/13 2024-02-06T21:30:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 2/2 2024-02-06T21:30:58.811 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 1/2 2024-02-06T21:30:59.113 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-06T21:30:59.114 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:59.114 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:30:59.114 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-06T21:30:59.114 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:30:59.114 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:30:59.392 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-volume 2024-02-06T21:30:59.605 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1269.g633ab857.el9.noarch 13/13 2024-02-06T21:30:59.606 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1269.g633ab857.el9.noarch 1/13 2024-02-06T21:30:59.606 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-06T21:30:59.606 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-06T21:30:59.606 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-06T21:30:59.606 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-06T21:30:59.606 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-06T21:30:59.606 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-06T21:30:59.606 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-06T21:30:59.606 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-06T21:30:59.606 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-06T21:30:59.606 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-06T21:30:59.606 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-06T21:30:59.983 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:35 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:30:59.986 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-06T21:30:59.986 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:59.986 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:30:59.987 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:30:59.987 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-06T21:30:59.987 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-06T21:30:59.987 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-06T21:30:59.987 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-06T21:30:59.987 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-06T21:30:59.987 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-06T21:30:59.987 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-06T21:30:59.988 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-06T21:30:59.988 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-06T21:30:59.988 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-06T21:30:59.988 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-06T21:30:59.988 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-06T21:30:59.988 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:30:59.988 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:00.139 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:31:00.140 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:00.140 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-02-06T21:31:00.140 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:00.141 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:31:00.141 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 263 k 2024-02-06T21:31:00.141 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-02-06T21:31:00.141 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-06T21:31:00.141 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:00.141 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:31:00.141 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:00.141 INFO:teuthology.orchestra.run.smithi120.stdout:Install 2 Packages 2024-02-06T21:31:00.142 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:00.142 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 332 k 2024-02-06T21:31:00.142 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 1.5 M 2024-02-06T21:31:00.142 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:31:00.260 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-cephadm 2024-02-06T21:31:00.261 INFO:teuthology.orchestra.run.smithi120.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 572 kB/s | 69 kB 00:00 2024-02-06T21:31:00.420 INFO:teuthology.orchestra.run.smithi120.stdout:(2/2): ceph-volume-19.0.0-1269.g633ab857.el9.no 943 kB/s | 263 kB 00:00 2024-02-06T21:31:00.421 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:00.422 INFO:teuthology.orchestra.run.smithi120.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-02-06T21:31:00.422 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:31:00.430 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:31:00.430 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:31:00.471 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:31:00.472 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:31:00.622 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:31:00.801 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-02-06T21:31:00.832 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:21 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:00.839 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-volume-2:19.0.0-1269.g633ab857.el9.noarch 2/2 2024-02-06T21:31:00.855 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-volume-2:19.0.0-1269.g633ab857.el9.noarch 2/2 2024-02-06T21:31:00.855 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:31:00.855 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-06T21:31:00.855 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:00.991 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:00.992 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2024-02-06T21:31:00.992 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-02-06T21:31:00.992 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2024-02-06T21:31:00.992 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:31:00.992 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 138 k 2024-02-06T21:31:00.992 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-02-06T21:31:00.992 INFO:teuthology.orchestra.run.smithi135.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-06T21:31:00.992 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-06T21:31:00.992 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-06T21:31:00.992 INFO:teuthology.orchestra.run.smithi135.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-06T21:31:00.992 INFO:teuthology.orchestra.run.smithi135.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-06T21:31:00.992 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:00.992 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:31:00.993 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2024-02-06T21:31:00.993 INFO:teuthology.orchestra.run.smithi135.stdout:Install 6 Packages 2024-02-06T21:31:00.993 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:00.993 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 6.8 M 2024-02-06T21:31:00.993 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 33 M 2024-02-06T21:31:00.993 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:31:01.273 INFO:teuthology.orchestra.run.smithi135.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.1 MB/s | 227 kB 00:00 2024-02-06T21:31:01.315 INFO:teuthology.orchestra.run.smithi135.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1269.g633ab857.e 547 kB/s | 138 kB 00:00 2024-02-06T21:31:01.382 INFO:teuthology.orchestra.run.smithi135.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 876 kB/s | 58 kB 00:00 2024-02-06T21:31:01.407 INFO:teuthology.orchestra.run.smithi135.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 4.0 MB/s | 548 kB 00:00 2024-02-06T21:31:01.432 INFO:teuthology.orchestra.run.smithi135.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-02-06T21:31:01.515 INFO:teuthology.orchestra.run.smithi135.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2024-02-06T21:31:01.517 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:01.518 INFO:teuthology.orchestra.run.smithi135.stdout:Total 13 MB/s | 6.8 MB 00:00 2024-02-06T21:31:01.518 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:31:01.526 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:31:01.527 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:31:01.601 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-volume-2:19.0.0-1269.g633ab857.el9.noarch 1/2 2024-02-06T21:31:01.616 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:31:01.617 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:31:01.937 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:31:01.952 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-02-06T21:31:01.952 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:01.953 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:31:01.953 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:31:01.953 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-06T21:31:01.953 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:01.953 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:31:02.104 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-06T21:31:02.158 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-06T21:31:02.281 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install librados-devel 2024-02-06T21:31:02.669 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-06T21:31:02.761 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-06T21:31:02.823 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-06T21:31:02.846 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1269.g633ab857.el9.noarch 6/6 2024-02-06T21:31:02.885 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:38 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:31:03.020 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:31:03.021 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:03.021 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:31:03.021 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:03.021 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:31:03.021 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel x86_64 2:19.0.0-1269.g633ab857.el9 ceph 127 k 2024-02-06T21:31:03.021 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:03.021 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:31:03.021 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:03.022 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:31:03.022 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:03.022 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 127 k 2024-02-06T21:31:03.022 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 456 k 2024-02-06T21:31:03.022 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:31:03.254 INFO:teuthology.orchestra.run.smithi120.stdout:librados-devel-19.0.0-1269.g633ab857.el9.x86_64 546 kB/s | 127 kB 00:00 2024-02-06T21:31:03.255 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:03.255 INFO:teuthology.orchestra.run.smithi120.stdout:Total 542 kB/s | 127 kB 00:00 2024-02-06T21:31:03.255 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:31:03.260 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:31:03.260 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:31:03.315 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:31:03.315 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:31:03.465 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:31:03.553 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados-devel-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:03.694 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1269.g633ab857.el9.noarch 6/6 2024-02-06T21:31:03.694 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1269.g633ab857.el9.noarch 1/6 2024-02-06T21:31:03.694 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-06T21:31:03.694 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-06T21:31:03.694 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-06T21:31:03.694 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-06T21:31:04.020 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados-devel-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:04.023 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-06T21:31:04.024 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:04.024 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:31:04.024 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:31:04.024 INFO:teuthology.orchestra.run.smithi135.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-06T21:31:04.024 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-06T21:31:04.024 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-06T21:31:04.024 INFO:teuthology.orchestra.run.smithi135.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-06T21:31:04.024 INFO:teuthology.orchestra.run.smithi135.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-06T21:31:04.024 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:04.024 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:04.265 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-fuse 2024-02-06T21:31:04.352 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados-devel-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:04.352 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:04.353 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:31:04.353 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:31:04.353 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:04.353 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:31:04.608 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install libcephfs2 2024-02-06T21:31:04.832 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:25 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:04.993 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:04.994 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:04.994 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-02-06T21:31:04.994 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:04.994 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:31:04.994 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse x86_64 2:19.0.0-1269.g633ab857.el9 ceph 886 k 2024-02-06T21:31:04.994 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-02-06T21:31:04.994 INFO:teuthology.orchestra.run.smithi135.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-06T21:31:04.994 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:04.995 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:31:04.995 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:04.995 INFO:teuthology.orchestra.run.smithi135.stdout:Install 2 Packages 2024-02-06T21:31:04.995 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:04.995 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 964 k 2024-02-06T21:31:04.995 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 2.8 M 2024-02-06T21:31:04.996 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:31:05.142 INFO:teuthology.orchestra.run.smithi135.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 537 kB/s | 79 kB 00:00 2024-02-06T21:31:05.189 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:41 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:31:05.295 INFO:teuthology.orchestra.run.smithi120.stdout:Package libcephfs2-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:31:05.308 INFO:teuthology.orchestra.run.smithi135.stdout:(2/2): ceph-fuse-19.0.0-1269.g633ab857.el9.x86_ 2.8 MB/s | 886 kB 00:00 2024-02-06T21:31:05.310 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:05.310 INFO:teuthology.orchestra.run.smithi135.stdout:Total 3.0 MB/s | 964 kB 00:00 2024-02-06T21:31:05.310 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:31:05.331 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:31:05.331 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:31:05.347 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:31:05.348 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-02-06T21:31:05.348 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:31:05.407 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:31:05.407 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:31:05.422 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install libcephfs-devel 2024-02-06T21:31:05.590 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:31:05.735 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-06T21:31:05.800 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 2/2 2024-02-06T21:31:05.991 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:41 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:31:06.149 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:31:06.150 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:06.150 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:31:06.150 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:06.150 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:31:06.150 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel x86_64 2:19.0.0-1269.g633ab857.el9 ceph 31 k 2024-02-06T21:31:06.150 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:06.150 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:31:06.150 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:06.151 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:31:06.151 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:06.151 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 31 k 2024-02-06T21:31:06.151 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 140 k 2024-02-06T21:31:06.151 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:31:06.318 INFO:teuthology.orchestra.run.smithi120.stdout:libcephfs-devel-19.0.0-1269.g633ab857.el9.x86_6 187 kB/s | 31 kB 00:00 2024-02-06T21:31:06.318 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:06.318 INFO:teuthology.orchestra.run.smithi120.stdout:Total 186 kB/s | 31 kB 00:00 2024-02-06T21:31:06.318 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:31:06.322 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:31:06.322 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:31:06.340 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:31:06.341 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:31:06.423 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:31:06.541 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs-devel-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:06.577 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 2/2 2024-02-06T21:31:06.577 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 1/2 2024-02-06T21:31:06.907 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-06T21:31:06.908 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:06.908 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:31:06.908 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-06T21:31:06.908 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:06.908 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:06.910 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:07.145 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-volume 2024-02-06T21:31:07.170 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs-devel-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:07.170 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:07.170 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:31:07.171 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:31:07.171 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:07.171 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:31:07.356 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install librados2 2024-02-06T21:31:07.734 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:28 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:07.891 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:07.891 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:07.891 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-02-06T21:31:07.891 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:07.892 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:31:07.892 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-volume noarch 2:19.0.0-1269.g633ab857.el9 ceph-noarch 263 k 2024-02-06T21:31:07.892 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-02-06T21:31:07.892 INFO:teuthology.orchestra.run.smithi135.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-06T21:31:07.892 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:07.892 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:31:07.892 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:07.892 INFO:teuthology.orchestra.run.smithi135.stdout:Install 2 Packages 2024-02-06T21:31:07.892 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:07.893 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 332 k 2024-02-06T21:31:07.893 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 1.5 M 2024-02-06T21:31:07.893 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:31:07.930 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:43 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:31:08.035 INFO:teuthology.orchestra.run.smithi120.stdout:Package librados2-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:31:08.087 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:31:08.088 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-02-06T21:31:08.088 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:31:08.129 INFO:teuthology.orchestra.run.smithi135.stdout:(1/2): ceph-volume-19.0.0-1269.g633ab857.el9.no 1.1 MB/s | 263 kB 00:00 2024-02-06T21:31:08.162 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install librbd1 2024-02-06T21:31:08.163 INFO:teuthology.orchestra.run.smithi135.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 255 kB/s | 69 kB 00:00 2024-02-06T21:31:08.164 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:08.164 INFO:teuthology.orchestra.run.smithi135.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-02-06T21:31:08.164 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:31:08.173 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:31:08.173 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:31:08.212 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:31:08.212 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:31:08.360 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:31:08.536 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-02-06T21:31:08.590 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-volume-2:19.0.0-1269.g633ab857.el9.noarch 2/2 2024-02-06T21:31:08.606 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-volume-2:19.0.0-1269.g633ab857.el9.noarch 2/2 2024-02-06T21:31:08.606 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:31:08.607 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-06T21:31:08.607 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:08.728 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:44 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:31:08.831 INFO:teuthology.orchestra.run.smithi120.stdout:Package librbd1-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:31:08.884 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:31:08.885 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-02-06T21:31:08.885 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:31:08.953 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rados 2024-02-06T21:31:09.351 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-volume-2:19.0.0-1269.g633ab857.el9.noarch 1/2 2024-02-06T21:31:09.524 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:45 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:31:09.627 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-rados-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:31:09.679 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:31:09.680 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-02-06T21:31:09.680 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:31:09.697 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-02-06T21:31:09.698 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:09.698 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:31:09.698 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-volume-2:19.0.0-1269.g633ab857.el9.noarch 2024-02-06T21:31:09.698 INFO:teuthology.orchestra.run.smithi135.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-06T21:31:09.698 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:09.698 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:09.753 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rgw 2024-02-06T21:31:10.022 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados-devel 2024-02-06T21:31:10.323 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:46 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:31:10.426 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-rgw-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:31:10.478 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:31:10.478 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-02-06T21:31:10.478 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:31:10.554 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-cephfs 2024-02-06T21:31:10.610 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:31 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:10.769 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:10.770 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:10.770 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:31:10.770 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:10.770 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:31:10.770 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel x86_64 2:19.0.0-1269.g633ab857.el9 ceph 127 k 2024-02-06T21:31:10.770 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:10.770 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:31:10.770 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:10.770 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:31:10.770 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:10.770 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 127 k 2024-02-06T21:31:10.770 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 456 k 2024-02-06T21:31:10.771 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:31:11.094 INFO:teuthology.orchestra.run.smithi135.stdout:librados-devel-19.0.0-1269.g633ab857.el9.x86_64 392 kB/s | 127 kB 00:00 2024-02-06T21:31:11.095 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:11.095 INFO:teuthology.orchestra.run.smithi135.stdout:Total 391 kB/s | 127 kB 00:00 2024-02-06T21:31:11.095 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:31:11.100 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:31:11.100 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:31:11.117 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:47 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:31:11.156 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:31:11.156 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:31:11.222 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-cephfs-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:31:11.275 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:31:11.276 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-02-06T21:31:11.276 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:31:11.294 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:31:11.345 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rbd 2024-02-06T21:31:11.417 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librados-devel-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:11.908 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados-devel-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:11.912 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:47 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:31:12.015 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-rbd-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:31:12.070 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:31:12.071 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-02-06T21:31:12.071 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:31:12.144 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-fuse 2024-02-06T21:31:12.223 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados-devel-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:12.224 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:12.224 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:31:12.224 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:31:12.224 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:12.224 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:12.470 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install libcephfs2 2024-02-06T21:31:12.727 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:48 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:31:12.894 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:31:12.894 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:12.895 INFO:teuthology.orchestra.run.smithi120.stdout: Package Architecture Version Repository Size 2024-02-06T21:31:12.895 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:12.895 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:31:12.895 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse x86_64 2:19.0.0-1269.g633ab857.el9 ceph 85 k 2024-02-06T21:31:12.895 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:12.895 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:31:12.895 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:12.895 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:31:12.895 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:12.895 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 85 k 2024-02-06T21:31:12.895 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 231 k 2024-02-06T21:31:12.895 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:31:13.050 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:34 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:13.105 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-fuse-19.0.0-1269.g633ab857.el9.x86_64.rpm 408 kB/s | 85 kB 00:00 2024-02-06T21:31:13.105 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:13.106 INFO:teuthology.orchestra.run.smithi120.stdout:Total 405 kB/s | 85 kB 00:00 2024-02-06T21:31:13.106 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:31:13.113 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:31:13.113 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:31:13.154 INFO:teuthology.orchestra.run.smithi135.stdout:Package libcephfs2-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:31:13.174 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:31:13.174 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:31:13.206 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:13.206 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-02-06T21:31:13.207 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:13.271 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install libcephfs-devel 2024-02-06T21:31:13.312 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:31:13.413 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:13.816 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:13.844 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:34 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:14.001 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:14.002 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:14.002 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:31:14.002 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:14.002 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:31:14.003 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel x86_64 2:19.0.0-1269.g633ab857.el9 ceph 31 k 2024-02-06T21:31:14.003 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:14.003 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:31:14.003 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:14.003 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:31:14.003 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:14.003 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 31 k 2024-02-06T21:31:14.003 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 140 k 2024-02-06T21:31:14.003 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:31:14.118 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:14.119 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:14.119 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:31:14.119 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:31:14.119 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:14.119 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:31:14.174 INFO:teuthology.orchestra.run.smithi135.stdout:libcephfs-devel-19.0.0-1269.g633ab857.el9.x86_6 183 kB/s | 31 kB 00:00 2024-02-06T21:31:14.174 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:14.174 INFO:teuthology.orchestra.run.smithi135.stdout:Total 182 kB/s | 31 kB 00:00 2024-02-06T21:31:14.175 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:31:14.178 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:31:14.178 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:31:14.220 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:31:14.220 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:31:14.303 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:31:14.325 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-mirror 2024-02-06T21:31:14.421 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs-devel-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:14.858 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:14.907 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:50 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:31:15.068 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:31:15.069 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:15.069 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:31:15.069 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:15.069 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:31:15.069 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror x86_64 2:19.0.0-1269.g633ab857.el9 ceph 3.1 M 2024-02-06T21:31:15.069 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:15.069 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:31:15.069 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:15.069 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:31:15.069 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:15.070 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 3.1 M 2024-02-06T21:31:15.070 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 13 M 2024-02-06T21:31:15.070 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:31:15.149 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs-devel-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:15.150 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:31:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:31:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:15.150 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:15.337 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados2 2024-02-06T21:31:15.451 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-mirror-19.0.0-1269.g633ab857.el9.x86_64.rpm 8.1 MB/s | 3.1 MB 00:00 2024-02-06T21:31:15.452 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:15.452 INFO:teuthology.orchestra.run.smithi120.stdout:Total 8.1 MB/s | 3.1 MB 00:00 2024-02-06T21:31:15.452 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:31:15.461 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:31:15.461 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:31:15.534 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:31:15.534 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:31:15.910 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:36 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:15.958 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:31:16.014 INFO:teuthology.orchestra.run.smithi135.stdout:Package librados2-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:31:16.018 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-mirror-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:16.043 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:16.044 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:31:16.044 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-06T21:31:16.044 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-06T21:31:16.044 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-06T21:31:16.044 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:16.067 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:16.068 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-02-06T21:31:16.069 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:16.136 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librbd1 2024-02-06T21:31:16.705 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:37 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:16.808 INFO:teuthology.orchestra.run.smithi135.stdout:Package librbd1-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:31:16.861 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:16.862 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-02-06T21:31:16.862 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:16.925 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rados 2024-02-06T21:31:17.233 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-mirror-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:17.233 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:17.233 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:31:17.233 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:31:17.233 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:17.233 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:31:17.480 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-nbd 2024-02-06T21:31:17.498 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:38 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:17.601 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rados-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:31:17.654 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:17.654 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-02-06T21:31:17.655 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:17.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rgw 2024-02-06T21:31:18.063 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:54 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:31:18.222 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:31:18.223 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:18.223 INFO:teuthology.orchestra.run.smithi120.stdout: Package Architecture Version Repository Size 2024-02-06T21:31:18.223 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:18.223 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:31:18.223 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd x86_64 2:19.0.0-1269.g633ab857.el9 ceph 171 k 2024-02-06T21:31:18.223 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:18.223 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:31:18.223 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:18.223 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:31:18.223 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:18.223 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 171 k 2024-02-06T21:31:18.223 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 494 k 2024-02-06T21:31:18.223 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:31:18.299 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:39 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:18.402 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rgw-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:31:18.455 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:18.456 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-02-06T21:31:18.456 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:18.533 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-cephfs 2024-02-06T21:31:18.581 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-nbd-19.0.0-1269.g633ab857.el9.x86_64.rpm 478 kB/s | 171 kB 00:00 2024-02-06T21:31:18.581 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:18.582 INFO:teuthology.orchestra.run.smithi120.stdout:Total 476 kB/s | 171 kB 00:00 2024-02-06T21:31:18.582 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:31:18.590 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:31:18.590 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:31:18.645 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:31:18.646 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:31:18.851 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:31:18.955 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-nbd-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:19.102 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:40 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:19.207 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-cephfs-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:31:19.262 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:19.263 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-02-06T21:31:19.263 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:19.323 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rbd 2024-02-06T21:31:19.405 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:19.689 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-nbd-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:19.689 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:19.689 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:31:19.689 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:31:19.689 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:19.689 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:31:19.892 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:40 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:19.965 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-base-debuginfo 2024-02-06T21:31:19.995 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rbd-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:31:20.047 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:20.048 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-02-06T21:31:20.048 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:20.112 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-fuse 2024-02-06T21:31:20.552 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:56 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:31:20.685 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:41 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:20.710 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:31:20.711 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:20.711 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:31:20.711 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:20.711 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:31:20.711 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 67 M 2024-02-06T21:31:20.711 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-02-06T21:31:20.711 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 22 M 2024-02-06T21:31:20.711 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:20.711 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:31:20.711 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:20.711 INFO:teuthology.orchestra.run.smithi120.stdout:Install 2 Packages 2024-02-06T21:31:20.711 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:20.712 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 89 M 2024-02-06T21:31:20.712 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 493 M 2024-02-06T21:31:20.712 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:31:20.844 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:20.845 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2024-02-06T21:31:20.845 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:20.845 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:31:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse x86_64 2:19.0.0-1269.g633ab857.el9 ceph 85 k 2024-02-06T21:31:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:20.845 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:31:20.845 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:20.845 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:31:20.846 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:20.846 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 85 k 2024-02-06T21:31:20.846 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 231 k 2024-02-06T21:31:20.846 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:31:21.185 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-fuse-19.0.0-1269.g633ab857.el9.x86_64.rpm 252 kB/s | 85 kB 00:00 2024-02-06T21:31:21.185 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:21.185 INFO:teuthology.orchestra.run.smithi135.stdout:Total 251 kB/s | 85 kB 00:00 2024-02-06T21:31:21.185 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:31:21.192 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:31:21.193 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:31:21.247 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:31:21.247 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:31:21.390 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:31:21.500 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:21.928 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:22.240 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:22.240 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:22.240 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:31:22.240 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:31:22.240 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:22.240 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:22.469 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-mirror 2024-02-06T21:31:23.052 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:44 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:23.193 INFO:teuthology.orchestra.run.smithi120.stdout:(1/2): ceph-debuginfo-19.0.0-1269.g633ab857.el9 9.0 MB/s | 22 MB 00:02 2024-02-06T21:31:23.214 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:23.215 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:23.215 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:31:23.215 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:23.215 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:31:23.215 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror x86_64 2:19.0.0-1269.g633ab857.el9 ceph 3.1 M 2024-02-06T21:31:23.215 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:23.215 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:31:23.215 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:23.215 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:31:23.215 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:23.216 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 3.1 M 2024-02-06T21:31:23.216 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 13 M 2024-02-06T21:31:23.216 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:31:23.613 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-mirror-19.0.0-1269.g633ab857.el9.x86_64.rpm 7.8 MB/s | 3.1 MB 00:00 2024-02-06T21:31:23.614 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:23.614 INFO:teuthology.orchestra.run.smithi135.stdout:Total 7.8 MB/s | 3.1 MB 00:00 2024-02-06T21:31:23.614 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:31:23.623 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:31:23.623 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:31:23.696 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:31:23.696 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:31:24.122 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:31:24.180 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-mirror-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:24.205 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:24.205 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-06T21:31:24.205 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-06T21:31:24.205 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-06T21:31:24.205 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-06T21:31:24.206 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:25.345 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-mirror-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:25.345 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:25.345 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:31:25.345 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:31:25.345 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:25.345 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:25.576 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-nbd 2024-02-06T21:31:26.162 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:47 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:26.320 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:26.321 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:26.321 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2024-02-06T21:31:26.321 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:26.321 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:31:26.321 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd x86_64 2:19.0.0-1269.g633ab857.el9 ceph 171 k 2024-02-06T21:31:26.321 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:26.321 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:31:26.321 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:26.321 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:31:26.322 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:26.322 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 171 k 2024-02-06T21:31:26.322 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 494 k 2024-02-06T21:31:26.322 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:31:26.549 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-nbd-19.0.0-1269.g633ab857.el9.x86_64.rpm 750 kB/s | 171 kB 00:00 2024-02-06T21:31:26.550 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:26.550 INFO:teuthology.orchestra.run.smithi135.stdout:Total 746 kB/s | 171 kB 00:00 2024-02-06T21:31:26.550 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:31:26.558 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:31:26.558 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:31:26.613 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:31:26.614 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:31:26.817 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:31:26.923 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-nbd-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:27.402 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:27.717 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-nbd-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:31:27.718 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:27.718 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:31:27.718 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:31:27.718 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:27.718 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:27.934 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-base-debuginfo 2024-02-06T21:31:28.527 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:49 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:28.698 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:28.698 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:28.698 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:31:28.699 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:28.699 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:31:28.699 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 67 M 2024-02-06T21:31:28.699 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-02-06T21:31:28.699 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 22 M 2024-02-06T21:31:28.699 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:28.699 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:31:28.699 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:28.699 INFO:teuthology.orchestra.run.smithi135.stdout:Install 2 Packages 2024-02-06T21:31:28.699 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:28.700 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 89 M 2024-02-06T21:31:28.700 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 493 M 2024-02-06T21:31:28.700 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:31:33.178 INFO:teuthology.orchestra.run.smithi135.stdout:(1/2): ceph-base-debuginfo-19.0.0-1269.g633ab85 15 MB/s | 67 MB 00:04 2024-02-06T21:31:33.944 INFO:teuthology.orchestra.run.smithi135.stdout:(2/2): ceph-debuginfo-19.0.0-1269.g633ab857.el9 4.3 MB/s | 22 MB 00:05 2024-02-06T21:31:33.946 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:33.947 INFO:teuthology.orchestra.run.smithi135.stdout:Total 17 MB/s | 89 MB 00:05 2024-02-06T21:31:33.947 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:31:33.952 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:31:33.952 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:31:34.556 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:31:34.557 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:31:34.700 INFO:teuthology.orchestra.run.smithi120.stdout:(2/2): ceph-base-debuginfo-19.0.0-1269.g633ab85 4.8 MB/s | 67 MB 00:13 2024-02-06T21:31:34.703 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:31:34.703 INFO:teuthology.orchestra.run.smithi120.stdout:Total 6.4 MB/s | 89 MB 00:13 2024-02-06T21:31:34.703 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:31:34.708 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:31:34.708 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:31:35.298 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:31:35.298 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:31:37.639 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:31:38.387 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:31:44.607 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 1/2 2024-02-06T21:31:44.678 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 2/2 2024-02-06T21:31:45.365 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 1/2 2024-02-06T21:31:45.444 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 2/2 2024-02-06T21:31:48.213 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 2/2 2024-02-06T21:31:48.213 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/2 2024-02-06T21:31:48.507 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2/2 2024-02-06T21:31:48.508 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:48.508 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:31:48.508 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:31:48.508 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:31:48.508 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:48.508 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:31:48.722 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-common-debuginfo 2024-02-06T21:31:48.953 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 2/2 2024-02-06T21:31:48.982 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/2 2024-02-06T21:31:49.215 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2/2 2024-02-06T21:31:49.215 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:49.215 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:31:49.215 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:31:49.215 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:31:49.215 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:49.215 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:31:49.298 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:10 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:31:49.438 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-common-debuginfo 2024-02-06T21:31:49.452 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:31:49.453 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:49.453 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:31:49.453 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:49.453 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:31:49.453 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 416 M 2024-02-06T21:31:49.453 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:49.453 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:31:49.453 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:31:49.453 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:31:49.454 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:31:49.454 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 416 M 2024-02-06T21:31:49.454 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 1.7 G 2024-02-06T21:31:49.454 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:31:50.036 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:02:26 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:31:50.194 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:31:50.195 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:50.195 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:31:50.195 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:50.195 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:31:50.195 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 416 M 2024-02-06T21:31:50.195 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:50.195 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:31:50.195 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:31:50.195 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:31:50.195 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:31:50.195 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 416 M 2024-02-06T21:31:50.196 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 1.7 G 2024-02-06T21:31:50.196 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:32:19.738 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-common-debuginfo-19.0.0-1269.g633ab857.el9 14 MB/s | 416 MB 00:30 2024-02-06T21:32:19.739 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:32:19.739 INFO:teuthology.orchestra.run.smithi135.stdout:Total 14 MB/s | 416 MB 00:30 2024-02-06T21:32:19.739 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:32:19.743 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:32:19.743 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:32:22.210 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:32:22.211 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:32:23.703 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-common-debuginfo-19.0.0-1269.g633ab857.el9 12 MB/s | 416 MB 00:33 2024-02-06T21:32:23.703 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:32:23.704 INFO:teuthology.orchestra.run.smithi120.stdout:Total 12 MB/s | 416 MB 00:33 2024-02-06T21:32:23.704 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:32:23.708 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:32:23.708 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:32:26.178 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:32:26.178 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:33:04.491 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:33:04.606 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:33:08.316 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:33:08.405 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:33:13.156 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:33:13.444 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:33:13.444 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:33:13.444 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:33:13.444 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:33:13.444 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:33:13.444 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:33:13.698 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-02-06T21:33:14.281 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:03:35 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:33:14.451 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:33:14.451 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2024-02-06T21:33:14.451 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:33:14.451 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2024-02-06T21:33:14.451 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:33:14.452 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 1.4 M 2024-02-06T21:33:14.452 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:33:14.452 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:33:14.452 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2024-02-06T21:33:14.452 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:33:14.452 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:33:14.452 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 1.4 M 2024-02-06T21:33:14.453 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 6.4 M 2024-02-06T21:33:14.453 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:33:15.361 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-12 1.6 MB/s | 1.4 MB 00:00 2024-02-06T21:33:15.361 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:33:15.362 INFO:teuthology.orchestra.run.smithi135.stdout:Total 1.6 MB/s | 1.4 MB 00:00 2024-02-06T21:33:15.362 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:33:15.365 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:33:15.365 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:33:15.406 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:33:15.407 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:33:15.658 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:33:15.767 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1269. 1/1 2024-02-06T21:33:16.220 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1269. 1/1 2024-02-06T21:33:16.503 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1269. 1/1 2024-02-06T21:33:16.503 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:33:16.503 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:33:16.503 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:33:16.503 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:33:16.503 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:33:16.683 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-radosgw-debuginfo 2024-02-06T21:33:16.770 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:33:17.038 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:33:17.038 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:33:17.038 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:33:17.039 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:33:17.039 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:33:17.039 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:33:17.251 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:03:38 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:33:17.276 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-02-06T21:33:17.407 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:33:17.408 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:33:17.409 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:33:17.409 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:33:17.409 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:33:17.409 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 176 M 2024-02-06T21:33:17.409 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:33:17.409 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:33:17.409 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:33:17.409 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:33:17.409 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:33:17.409 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 176 M 2024-02-06T21:33:17.409 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 753 M 2024-02-06T21:33:17.410 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:33:17.846 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:03:53 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:33:18.001 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:33:18.002 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-02-06T21:33:18.002 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:33:18.002 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-02-06T21:33:18.002 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:33:18.002 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 1.4 M 2024-02-06T21:33:18.003 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:33:18.003 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:33:18.003 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-02-06T21:33:18.003 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:33:18.003 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:33:18.003 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 1.4 M 2024-02-06T21:33:18.003 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 6.4 M 2024-02-06T21:33:18.003 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:33:18.318 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-12 4.5 MB/s | 1.4 MB 00:00 2024-02-06T21:33:18.318 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:33:18.319 INFO:teuthology.orchestra.run.smithi120.stdout:Total 4.5 MB/s | 1.4 MB 00:00 2024-02-06T21:33:18.319 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:33:18.322 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:33:18.322 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:33:18.363 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:33:18.363 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:33:18.616 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:33:18.716 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1269. 1/1 2024-02-06T21:33:19.102 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1269. 1/1 2024-02-06T21:33:19.416 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1269. 1/1 2024-02-06T21:33:19.416 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:33:19.416 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:33:19.416 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:33:19.416 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:33:19.416 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:33:19.573 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw-debuginfo 2024-02-06T21:33:20.137 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:03:56 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:33:20.292 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:33:20.293 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:33:20.293 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:33:20.293 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:33:20.293 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:33:20.294 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 176 M 2024-02-06T21:33:20.294 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:33:20.294 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:33:20.294 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:33:20.294 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:33:20.294 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:33:20.294 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 176 M 2024-02-06T21:33:20.294 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 753 M 2024-02-06T21:33:20.294 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:33:27.630 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-radosgw-debuginfo-19.0.0-1269.g633ab857.el 17 MB/s | 176 MB 00:10 2024-02-06T21:33:27.630 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:33:27.631 INFO:teuthology.orchestra.run.smithi135.stdout:Total 17 MB/s | 176 MB 00:10 2024-02-06T21:33:27.631 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:33:27.635 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:33:27.635 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:33:28.693 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:33:28.694 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:33:33.978 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-radosgw-debuginfo-19.0.0-1269.g633ab857.el 13 MB/s | 176 MB 00:13 2024-02-06T21:33:33.979 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:33:33.980 INFO:teuthology.orchestra.run.smithi120.stdout:Total 13 MB/s | 176 MB 00:13 2024-02-06T21:33:33.980 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:33:33.984 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:33:33.984 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:33:35.045 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:33:35.045 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:33:46.644 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:33:46.742 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x 1/1 2024-02-06T21:33:51.866 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x 1/1 2024-02-06T21:33:52.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x 1/1 2024-02-06T21:33:52.157 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:33:52.157 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:33:52.157 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:33:52.157 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:33:52.157 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:33:52.373 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-test-debuginfo 2024-02-06T21:33:52.947 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:04:13 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:33:53.077 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:33:53.101 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:33:53.102 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:33:53.102 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:33:53.102 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:33:53.102 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:33:53.102 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 672 M 2024-02-06T21:33:53.102 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:33:53.102 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:33:53.102 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:33:53.102 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:33:53.102 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:33:53.103 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 672 M 2024-02-06T21:33:53.103 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 3.1 G 2024-02-06T21:33:53.103 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:33:53.176 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x 1/1 2024-02-06T21:33:58.068 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x 1/1 2024-02-06T21:33:58.337 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x 1/1 2024-02-06T21:33:58.337 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:33:58.337 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:33:58.337 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:33:58.337 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:33:58.337 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:33:58.532 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-test-debuginfo 2024-02-06T21:33:59.101 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:04:35 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:33:59.256 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:33:59.257 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:33:59.257 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:33:59.257 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:33:59.257 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:33:59.257 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 672 M 2024-02-06T21:33:59.257 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:33:59.257 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:33:59.257 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:33:59.257 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:33:59.257 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:33:59.257 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 672 M 2024-02-06T21:33:59.257 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 3.1 G 2024-02-06T21:33:59.257 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:34:38.251 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-test-debuginfo-19.0.0-1269.g633ab857.el9.x 17 MB/s | 672 MB 00:38 2024-02-06T21:34:38.252 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:34:38.252 INFO:teuthology.orchestra.run.smithi120.stdout:Total 17 MB/s | 672 MB 00:38 2024-02-06T21:34:38.252 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:34:38.257 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:34:38.257 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:34:42.232 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:34:42.232 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:34:57.023 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-test-debuginfo-19.0.0-1269.g633ab857.el9.x 11 MB/s | 672 MB 01:03 2024-02-06T21:34:57.023 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:34:57.024 INFO:teuthology.orchestra.run.smithi135.stdout:Total 11 MB/s | 672 MB 01:03 2024-02-06T21:34:57.024 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:34:57.030 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:34:57.030 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:35:00.984 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:35:00.984 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:35:53.421 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:35:53.784 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:36:03.196 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:36:03.480 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:36:03.480 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:03.480 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:36:03.480 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:36:03.480 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:03.480 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:36:03.769 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-base-debuginfo 2024-02-06T21:36:04.342 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:40 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:36:04.446 INFO:teuthology.orchestra.run.smithi120.stdout:Package ceph-base-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:36:04.497 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:36:04.498 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-02-06T21:36:04.498 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:36:04.570 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-debuginfo 2024-02-06T21:36:05.140 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:41 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:36:05.295 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:36:05.295 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:36:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:36:05.295 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:36:05.295 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:36:05.295 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 20 M 2024-02-06T21:36:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:05.296 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:36:05.296 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:36:05.296 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:36:05.296 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:05.296 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 20 M 2024-02-06T21:36:05.297 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 103 M 2024-02-06T21:36:05.297 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:36:06.772 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-mgr-debuginfo-19.0.0-1269.g633ab857.el9.x8 13 MB/s | 20 MB 00:01 2024-02-06T21:36:06.772 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:36:06.773 INFO:teuthology.orchestra.run.smithi120.stdout:Total 13 MB/s | 20 MB 00:01 2024-02-06T21:36:06.773 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:36:06.777 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:36:06.777 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:36:06.924 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:36:06.925 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:36:09.159 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:36:09.495 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:36:10.479 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:36:10.740 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:36:10.741 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:10.741 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:36:10.741 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:36:10.741 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:10.741 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:36:10.922 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mds-debuginfo 2024-02-06T21:36:11.493 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:47 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:36:11.646 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:36:11.647 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:36:11.647 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:36:11.647 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:36:11.647 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:36:11.647 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 48 M 2024-02-06T21:36:11.647 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:11.647 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:36:11.648 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:36:11.648 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:36:11.648 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:11.648 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 48 M 2024-02-06T21:36:11.648 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 174 M 2024-02-06T21:36:11.648 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:36:11.760 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:36:11.856 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:36:14.100 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-mds-debuginfo-19.0.0-1269.g633ab857.el9.x8 19 MB/s | 48 MB 00:02 2024-02-06T21:36:14.101 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:36:14.101 INFO:teuthology.orchestra.run.smithi120.stdout:Total 19 MB/s | 48 MB 00:02 2024-02-06T21:36:14.101 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:36:14.105 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:36:14.105 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:36:14.419 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:36:14.420 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:36:19.155 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:36:19.291 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:36:20.785 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:36:21.056 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:36:21.056 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:21.056 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:36:21.056 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:36:21.056 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:21.057 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:36:21.242 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mon-debuginfo 2024-02-06T21:36:21.810 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:57 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:36:21.965 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:36:21.966 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:36:21.966 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:36:21.966 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:36:21.966 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:36:21.966 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 99 M 2024-02-06T21:36:21.966 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:21.966 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:36:21.966 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:36:21.966 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:36:21.967 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:21.967 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 99 M 2024-02-06T21:36:21.967 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 449 M 2024-02-06T21:36:21.967 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:36:22.314 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:36:22.601 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:36:22.601 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:36:22.601 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:36:22.601 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:36:22.601 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:36:22.601 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:36:22.888 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-base-debuginfo 2024-02-06T21:36:23.465 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:06:44 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:36:23.569 INFO:teuthology.orchestra.run.smithi135.stdout:Package ceph-base-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 is already installed. 2024-02-06T21:36:23.621 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:36:23.622 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-02-06T21:36:23.623 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:36:23.694 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-debuginfo 2024-02-06T21:36:24.264 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:06:45 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:36:24.418 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:36:24.419 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:36:24.419 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:36:24.419 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:36:24.419 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:36:24.419 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 20 M 2024-02-06T21:36:24.419 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:36:24.419 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:36:24.419 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:36:24.419 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:36:24.420 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:36:24.420 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 20 M 2024-02-06T21:36:24.420 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 103 M 2024-02-06T21:36:24.420 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:36:28.017 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-mgr-debuginfo-19.0.0-1269.g633ab857.el9.x8 5.5 MB/s | 20 MB 00:03 2024-02-06T21:36:28.018 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:36:28.019 INFO:teuthology.orchestra.run.smithi135.stdout:Total 5.5 MB/s | 20 MB 00:03 2024-02-06T21:36:28.019 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:36:28.023 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:36:28.023 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:36:28.171 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:36:28.172 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:36:28.427 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-mon-debuginfo-19.0.0-1269.g633ab857.el9.x8 15 MB/s | 99 MB 00:06 2024-02-06T21:36:28.428 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:36:28.428 INFO:teuthology.orchestra.run.smithi120.stdout:Total 15 MB/s | 99 MB 00:06 2024-02-06T21:36:28.428 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:36:28.432 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:36:28.432 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:36:29.039 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:36:29.040 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:36:30.398 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:36:30.559 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:36:31.759 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:36:32.039 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:36:32.040 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:36:32.040 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:36:32.040 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:36:32.040 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:36:32.040 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:36:32.216 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mds-debuginfo 2024-02-06T21:36:32.790 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:06:53 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:36:32.946 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:36:32.946 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:36:32.947 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:36:32.947 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:36:32.947 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:36:32.947 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 48 M 2024-02-06T21:36:32.947 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:36:32.947 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:36:32.947 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:36:32.947 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:36:32.947 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:36:32.947 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 48 M 2024-02-06T21:36:32.947 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 174 M 2024-02-06T21:36:32.947 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:36:35.503 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-mds-debuginfo-19.0.0-1269.g633ab857.el9.x8 19 MB/s | 48 MB 00:02 2024-02-06T21:36:35.504 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:36:35.504 INFO:teuthology.orchestra.run.smithi135.stdout:Total 19 MB/s | 48 MB 00:02 2024-02-06T21:36:35.504 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:36:35.508 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:36:35.508 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:36:35.821 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:36:35.821 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:36:39.301 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:36:39.408 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:36:40.585 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:36:40.683 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:36:42.068 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:36:42.345 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:36:42.345 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:36:42.346 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:36:42.346 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:36:42.346 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:36:42.346 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:36:42.507 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mon-debuginfo 2024-02-06T21:36:42.508 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:36:42.793 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:36:42.793 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:36:42.793 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:36:42.793 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:36:42.972 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-osd-debuginfo 2024-02-06T21:36:43.078 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:07:04 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:36:43.232 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:36:43.233 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:36:43.233 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:36:43.233 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:36:43.233 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:36:43.233 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 99 M 2024-02-06T21:36:43.233 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:36:43.233 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:36:43.233 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:36:43.234 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:36:43.234 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:36:43.234 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 99 M 2024-02-06T21:36:43.234 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 449 M 2024-02-06T21:36:43.234 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:36:43.546 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:07:19 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:36:43.700 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:36:43.701 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:36:43.701 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:36:43.701 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:36:43.701 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:36:43.701 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 287 M 2024-02-06T21:36:43.701 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:43.701 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:36:43.702 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:36:43.702 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:36:43.702 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:36:43.702 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 287 M 2024-02-06T21:36:43.702 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 1.2 G 2024-02-06T21:36:43.702 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:36:57.987 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-mon-debuginfo-19.0.0-1269.g633ab857.el9.x8 6.7 MB/s | 99 MB 00:14 2024-02-06T21:36:57.988 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:36:57.988 INFO:teuthology.orchestra.run.smithi135.stdout:Total 6.7 MB/s | 99 MB 00:14 2024-02-06T21:36:57.989 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:36:57.992 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:36:57.993 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:36:58.601 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:36:58.601 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:37:08.859 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:37:08.969 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:37:12.159 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:37:12.451 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:37:12.451 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:37:12.451 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:37:12.451 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:37:12.451 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:37:12.451 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:37:12.635 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-osd-debuginfo 2024-02-06T21:37:13.204 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:07:34 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:37:13.357 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:37:13.358 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:37:13.358 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:37:13.359 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:37:13.359 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:37:13.359 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 287 M 2024-02-06T21:37:13.359 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:37:13.359 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:37:13.359 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:37:13.359 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:37:13.359 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:37:13.359 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 287 M 2024-02-06T21:37:13.359 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 1.2 G 2024-02-06T21:37:13.359 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:37:20.116 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-osd-debuginfo-19.0.0-1269.g633ab857.el9.x8 7.9 MB/s | 287 MB 00:36 2024-02-06T21:37:20.117 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:37:20.117 INFO:teuthology.orchestra.run.smithi120.stdout:Total 7.9 MB/s | 287 MB 00:36 2024-02-06T21:37:20.118 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:37:20.122 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:37:20.122 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:37:21.833 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:37:21.833 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:37:51.189 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:37:51.286 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:37:59.247 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:37:59.517 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:37:59.517 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:37:59.517 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:37:59.517 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:37:59.517 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:37:59.517 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:37:59.728 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-fuse-debuginfo 2024-02-06T21:38:00.302 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:08:36 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:38:00.455 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:38:00.456 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:00.456 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:38:00.456 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:00.456 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:38:00.456 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 5.9 M 2024-02-06T21:38:00.456 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:00.456 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:38:00.456 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:00.456 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:38:00.456 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:00.456 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 5.9 M 2024-02-06T21:38:00.457 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 28 M 2024-02-06T21:38:00.457 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:38:01.965 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-fuse-debuginfo-19.0.0-1269.g633ab857.el9.x 3.9 MB/s | 5.9 MB 00:01 2024-02-06T21:38:01.966 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:01.966 INFO:teuthology.orchestra.run.smithi120.stdout:Total 3.9 MB/s | 5.9 MB 00:01 2024-02-06T21:38:01.967 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:38:01.971 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:38:01.971 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:38:02.042 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:38:02.042 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:38:02.780 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:38:02.891 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:38:03.454 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:38:03.723 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:38:03.723 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:03.724 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:38:03.724 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:38:03.724 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:03.724 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:38:03.808 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-osd-debuginfo-19.0.0-1269.g633ab857.el9.x8 5.7 MB/s | 287 MB 00:50 2024-02-06T21:38:03.808 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:03.809 INFO:teuthology.orchestra.run.smithi135.stdout:Total 5.7 MB/s | 287 MB 00:50 2024-02-06T21:38:03.809 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:38:03.813 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:38:03.813 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:38:03.915 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install librados-devel-debuginfo 2024-02-06T21:38:04.483 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:08:40 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:38:04.640 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:38:04.640 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:04.640 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:38:04.641 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:04.641 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:38:04.641 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 109 k 2024-02-06T21:38:04.641 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:04.641 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:38:04.641 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:04.641 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:38:04.641 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:04.641 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 109 k 2024-02-06T21:38:04.641 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 440 k 2024-02-06T21:38:04.641 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:38:05.269 INFO:teuthology.orchestra.run.smithi120.stdout:librados-devel-debuginfo-19.0.0-1269.g633ab857. 173 kB/s | 109 kB 00:00 2024-02-06T21:38:05.269 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:05.270 INFO:teuthology.orchestra.run.smithi120.stdout:Total 173 kB/s | 109 kB 00:00 2024-02-06T21:38:05.270 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:38:05.273 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:38:05.273 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:38:05.310 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:38:05.310 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:38:05.430 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:38:05.525 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1269.g633ab857.el9 1/1 2024-02-06T21:38:05.531 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:38:05.531 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:38:05.926 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1269.g633ab857.el9 1/1 2024-02-06T21:38:06.214 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1269.g633ab857.el9 1/1 2024-02-06T21:38:06.214 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:06.215 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:38:06.215 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:38:06.215 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:06.215 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:38:06.385 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install libcephfs2-debuginfo 2024-02-06T21:38:06.950 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:08:42 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:38:07.105 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:38:07.106 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:07.106 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:38:07.106 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:07.106 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:38:07.106 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 5.4 M 2024-02-06T21:38:07.107 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:07.107 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:38:07.107 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:07.107 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:38:07.107 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:07.107 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 5.4 M 2024-02-06T21:38:07.107 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 25 M 2024-02-06T21:38:07.107 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:38:09.006 INFO:teuthology.orchestra.run.smithi120.stdout:libcephfs2-debuginfo-19.0.0-1269.g633ab857.el9. 2.8 MB/s | 5.4 MB 00:01 2024-02-06T21:38:09.006 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:09.007 INFO:teuthology.orchestra.run.smithi120.stdout:Total 2.8 MB/s | 5.4 MB 00:01 2024-02-06T21:38:09.007 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:38:09.011 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:38:09.011 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:38:09.079 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:38:09.079 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:38:09.751 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:38:09.845 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86 1/1 2024-02-06T21:38:10.390 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86 1/1 2024-02-06T21:38:10.662 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86 1/1 2024-02-06T21:38:10.662 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:10.663 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:38:10.663 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:38:10.663 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:10.663 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:38:10.835 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install librados2-debuginfo 2024-02-06T21:38:11.406 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:08:47 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:38:11.564 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:38:11.565 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:11.565 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:38:11.565 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:11.565 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:38:11.565 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 44 M 2024-02-06T21:38:11.565 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:11.565 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:38:11.565 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:11.565 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:38:11.565 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:11.565 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 44 M 2024-02-06T21:38:11.566 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 253 M 2024-02-06T21:38:11.566 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:38:14.525 INFO:teuthology.orchestra.run.smithi120.stdout:librados2-debuginfo-19.0.0-1269.g633ab857.el9.x 15 MB/s | 44 MB 00:02 2024-02-06T21:38:14.526 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:14.526 INFO:teuthology.orchestra.run.smithi120.stdout:Total 15 MB/s | 44 MB 00:02 2024-02-06T21:38:14.527 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:38:14.531 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:38:14.531 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:38:14.835 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:38:14.836 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:38:19.882 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:38:19.983 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:38:21.908 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:38:22.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:38:22.178 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:22.178 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:38:22.179 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:38:22.179 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:22.179 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:38:22.372 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install librbd1-debuginfo 2024-02-06T21:38:22.937 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:08:58 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:38:23.093 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:38:23.094 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:23.094 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:38:23.094 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:23.094 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:38:23.095 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 52 M 2024-02-06T21:38:23.095 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:23.095 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:38:23.095 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:23.095 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:38:23.095 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:23.095 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 52 M 2024-02-06T21:38:23.095 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 308 M 2024-02-06T21:38:23.095 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:38:26.011 INFO:teuthology.orchestra.run.smithi120.stdout:librbd1-debuginfo-19.0.0-1269.g633ab857.el9.x86 18 MB/s | 52 MB 00:02 2024-02-06T21:38:26.012 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:26.012 INFO:teuthology.orchestra.run.smithi120.stdout:Total 18 MB/s | 52 MB 00:02 2024-02-06T21:38:26.013 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:38:26.017 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:38:26.017 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:38:26.364 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:38:26.364 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:38:32.346 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:38:32.434 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librbd1-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:38:34.815 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:38:34.906 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:38:35.011 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:38:35.085 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:38:35.085 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:35.085 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:38:35.085 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:38:35.085 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:35.085 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:38:35.269 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-cephfs-debuginfo 2024-02-06T21:38:35.836 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:09:11 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:38:35.992 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:38:35.993 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:35.993 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:38:35.993 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:35.993 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:38:35.993 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 615 k 2024-02-06T21:38:35.993 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:35.993 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:38:35.993 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:35.993 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:38:35.993 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:35.993 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 615 k 2024-02-06T21:38:35.993 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 2.1 M 2024-02-06T21:38:35.993 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:38:36.306 INFO:teuthology.orchestra.run.smithi120.stdout:python3-cephfs-debuginfo-19.0.0-1269.g633ab857. 1.9 MB/s | 615 kB 00:00 2024-02-06T21:38:36.307 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:36.307 INFO:teuthology.orchestra.run.smithi120.stdout:Total 1.9 MB/s | 615 kB 00:00 2024-02-06T21:38:36.307 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:38:36.310 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:38:36.311 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:38:36.348 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:38:36.348 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:38:36.516 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:38:36.630 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1269.g633ab857.el9 1/1 2024-02-06T21:38:37.021 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1269.g633ab857.el9 1/1 2024-02-06T21:38:37.293 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1269.g633ab857.el9 1/1 2024-02-06T21:38:37.293 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:37.293 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:38:37.293 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:38:37.293 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:37.293 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:38:37.498 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rados-debuginfo 2024-02-06T21:38:38.067 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:09:14 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:38:38.224 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:38:38.225 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:38.225 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:38:38.225 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:38.225 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:38:38.225 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 770 k 2024-02-06T21:38:38.225 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:38.225 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:38:38.225 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:38.225 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:38:38.226 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:38.226 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 770 k 2024-02-06T21:38:38.226 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 2.7 M 2024-02-06T21:38:38.226 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:38:38.651 INFO:teuthology.orchestra.run.smithi120.stdout:python3-rados-debuginfo-19.0.0-1269.g633ab857.e 1.8 MB/s | 770 kB 00:00 2024-02-06T21:38:38.652 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:38.652 INFO:teuthology.orchestra.run.smithi120.stdout:Total 1.8 MB/s | 770 kB 00:00 2024-02-06T21:38:38.652 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:38:38.656 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:38:38.656 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:38:38.694 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:38:38.694 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:38:38.873 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:38:38.968 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1269.g633ab857.el9. 1/1 2024-02-06T21:38:39.394 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1269.g633ab857.el9. 1/1 2024-02-06T21:38:39.667 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1269.g633ab857.el9. 1/1 2024-02-06T21:38:39.668 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:39.668 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:38:39.668 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:38:39.668 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:39.668 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:38:39.877 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rbd-debuginfo 2024-02-06T21:38:40.452 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:09:16 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:38:40.607 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:38:40.608 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:40.608 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:38:40.608 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:40.608 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:38:40.608 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 1.1 M 2024-02-06T21:38:40.608 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:40.608 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:38:40.608 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:40.608 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:38:40.608 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:40.608 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 1.1 M 2024-02-06T21:38:40.608 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 4.3 M 2024-02-06T21:38:40.609 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:38:41.255 INFO:teuthology.orchestra.run.smithi120.stdout:python3-rbd-debuginfo-19.0.0-1269.g633ab857.el9 1.7 MB/s | 1.1 MB 00:00 2024-02-06T21:38:41.255 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:41.256 INFO:teuthology.orchestra.run.smithi120.stdout:Total 1.7 MB/s | 1.1 MB 00:00 2024-02-06T21:38:41.256 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:38:41.259 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:38:41.260 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:38:41.301 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:38:41.301 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:38:41.522 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:38:41.613 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:38:42.019 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:38:42.276 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:38:42.276 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:42.277 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:38:42.277 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:38:42.277 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:42.277 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:38:42.465 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rgw-debuginfo 2024-02-06T21:38:42.843 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:38:43.036 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:09:19 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:38:43.118 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:38:43.118 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:43.118 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:38:43.119 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:38:43.119 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:43.119 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:38:43.190 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:38:43.191 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:43.191 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:38:43.191 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:43.191 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:38:43.191 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 337 k 2024-02-06T21:38:43.191 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:43.191 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:38:43.191 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:43.191 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:38:43.192 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:43.192 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 337 k 2024-02-06T21:38:43.192 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 1.0 M 2024-02-06T21:38:43.192 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:38:43.328 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-fuse-debuginfo 2024-02-06T21:38:43.542 INFO:teuthology.orchestra.run.smithi120.stdout:python3-rgw-debuginfo-19.0.0-1269.g633ab857.el9 965 kB/s | 337 kB 00:00 2024-02-06T21:38:43.543 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:43.543 INFO:teuthology.orchestra.run.smithi120.stdout:Total 961 kB/s | 337 kB 00:00 2024-02-06T21:38:43.543 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:38:43.546 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:38:43.546 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:38:43.581 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:38:43.581 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:38:43.727 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:38:43.841 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:38:43.902 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:09:04 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:38:44.059 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:38:44.059 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:38:44.059 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:38:44.060 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:38:44.060 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:38:44.060 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 5.9 M 2024-02-06T21:38:44.060 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:44.060 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:38:44.060 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:38:44.060 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:38:44.060 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:44.060 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 5.9 M 2024-02-06T21:38:44.060 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 28 M 2024-02-06T21:38:44.060 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:38:44.231 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:38:44.500 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:38:44.500 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:44.500 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:38:44.500 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:38:44.500 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:44.500 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:38:44.701 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-fuse-debuginfo 2024-02-06T21:38:44.873 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-fuse-debuginfo-19.0.0-1269.g633ab857.el9.x 7.3 MB/s | 5.9 MB 00:00 2024-02-06T21:38:44.874 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:44.874 INFO:teuthology.orchestra.run.smithi135.stdout:Total 7.3 MB/s | 5.9 MB 00:00 2024-02-06T21:38:44.875 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:38:44.878 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:38:44.878 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:38:44.945 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:38:44.946 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:38:45.267 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:09:21 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:38:45.423 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:38:45.424 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:45.424 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:38:45.424 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:45.424 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:38:45.424 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 509 k 2024-02-06T21:38:45.424 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:45.424 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:38:45.424 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:45.424 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:38:45.424 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:45.424 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 509 k 2024-02-06T21:38:45.424 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 2.5 M 2024-02-06T21:38:45.425 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:38:45.690 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:38:45.806 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:38:45.912 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-fuse-debuginfo-19.0.0-1269.g633ab857.el9.x8 1.0 MB/s | 509 kB 00:00 2024-02-06T21:38:45.912 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:45.912 INFO:teuthology.orchestra.run.smithi120.stdout:Total 1.0 MB/s | 509 kB 00:00 2024-02-06T21:38:45.913 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:38:45.916 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:38:45.916 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:38:45.951 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:38:45.952 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:38:46.109 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:38:46.211 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:38:46.350 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:38:46.612 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:38:46.662 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:38:46.662 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:46.662 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:38:46.662 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:38:46.662 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:46.662 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:38:46.848 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados-devel-debuginfo 2024-02-06T21:38:46.864 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:38:46.864 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:46.865 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:38:46.865 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:38:46.865 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:46.865 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:38:47.042 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-mirror-debuginfo 2024-02-06T21:38:47.421 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:09:08 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:38:47.577 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:38:47.578 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:38:47.578 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:38:47.578 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:38:47.578 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:38:47.578 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 109 k 2024-02-06T21:38:47.578 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:47.578 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:38:47.578 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:38:47.578 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:38:47.578 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:47.578 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 109 k 2024-02-06T21:38:47.578 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 440 k 2024-02-06T21:38:47.579 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:38:47.609 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:09:23 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:38:47.763 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:38:47.764 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:47.764 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:38:47.764 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:47.764 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:38:47.764 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 52 M 2024-02-06T21:38:47.764 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:47.764 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:38:47.764 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:38:47.764 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:38:47.765 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:38:47.765 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 52 M 2024-02-06T21:38:47.765 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 298 M 2024-02-06T21:38:47.765 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:38:48.026 INFO:teuthology.orchestra.run.smithi135.stdout:librados-devel-debuginfo-19.0.0-1269.g633ab857. 243 kB/s | 109 kB 00:00 2024-02-06T21:38:48.027 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:48.027 INFO:teuthology.orchestra.run.smithi135.stdout:Total 242 kB/s | 109 kB 00:00 2024-02-06T21:38:48.027 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:38:48.030 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:38:48.030 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:38:48.066 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:38:48.066 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:38:48.185 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:38:48.284 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1269.g633ab857.el9 1/1 2024-02-06T21:38:48.691 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1269.g633ab857.el9 1/1 2024-02-06T21:38:48.995 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1269.g633ab857.el9 1/1 2024-02-06T21:38:48.995 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:48.995 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:38:48.995 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:38:48.995 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:48.995 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:38:49.181 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install libcephfs2-debuginfo 2024-02-06T21:38:49.754 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:09:10 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:38:49.911 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:38:49.912 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:38:49.912 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:38:49.912 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:38:49.912 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:38:49.912 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 5.4 M 2024-02-06T21:38:49.912 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:49.912 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:38:49.912 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:38:49.912 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:38:49.912 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:49.912 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 5.4 M 2024-02-06T21:38:49.912 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 25 M 2024-02-06T21:38:49.912 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:38:51.071 INFO:teuthology.orchestra.run.smithi135.stdout:libcephfs2-debuginfo-19.0.0-1269.g633ab857.el9. 4.6 MB/s | 5.4 MB 00:01 2024-02-06T21:38:51.071 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:51.072 INFO:teuthology.orchestra.run.smithi135.stdout:Total 4.6 MB/s | 5.4 MB 00:01 2024-02-06T21:38:51.072 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:38:51.076 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:38:51.076 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:38:51.143 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:38:51.143 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:38:51.823 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:38:51.936 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86 1/1 2024-02-06T21:38:52.553 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86 1/1 2024-02-06T21:38:52.844 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86 1/1 2024-02-06T21:38:52.844 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:52.844 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:38:52.844 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:38:52.844 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:52.844 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:38:53.037 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados2-debuginfo 2024-02-06T21:38:53.556 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-mirror-debuginfo-19.0.0-1269.g633ab857.el9. 8.9 MB/s | 52 MB 00:05 2024-02-06T21:38:53.557 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:53.558 INFO:teuthology.orchestra.run.smithi120.stdout:Total 8.9 MB/s | 52 MB 00:05 2024-02-06T21:38:53.558 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:38:53.562 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:38:53.562 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:38:53.610 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:09:14 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:38:53.768 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:38:53.769 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:38:53.769 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:38:53.769 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:38:53.769 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:38:53.769 INFO:teuthology.orchestra.run.smithi135.stdout: librados2-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 44 M 2024-02-06T21:38:53.769 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:53.769 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:38:53.769 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:38:53.769 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:38:53.769 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:38:53.769 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 44 M 2024-02-06T21:38:53.769 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 253 M 2024-02-06T21:38:53.770 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:38:53.899 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:38:53.899 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:38:56.107 INFO:teuthology.orchestra.run.smithi135.stdout:librados2-debuginfo-19.0.0-1269.g633ab857.el9.x 19 MB/s | 44 MB 00:02 2024-02-06T21:38:56.108 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:38:56.108 INFO:teuthology.orchestra.run.smithi135.stdout:Total 19 MB/s | 44 MB 00:02 2024-02-06T21:38:56.109 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:38:56.113 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:38:56.113 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:38:56.419 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:38:56.419 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:38:59.816 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:38:59.921 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1269.g633ab857.el9.x86 1/1 2024-02-06T21:39:01.618 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:39:01.714 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librados2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:39:02.115 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1269.g633ab857.el9.x86 1/1 2024-02-06T21:39:02.380 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1269.g633ab857.el9.x86 1/1 2024-02-06T21:39:02.380 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:02.380 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:39:02.380 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:39:02.380 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:02.380 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:39:02.559 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-nbd-debuginfo 2024-02-06T21:39:03.127 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:09:39 ago on Tue 06 Feb 2024 09:29:24 PM UTC. 2024-02-06T21:39:03.282 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-02-06T21:39:03.283 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:39:03.283 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-02-06T21:39:03.283 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:39:03.283 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-02-06T21:39:03.284 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 1.3 M 2024-02-06T21:39:03.284 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:03.284 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-02-06T21:39:03.284 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-02-06T21:39:03.284 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-02-06T21:39:03.284 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:03.284 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 1.3 M 2024-02-06T21:39:03.284 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 5.5 M 2024-02-06T21:39:03.284 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-02-06T21:39:03.592 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-nbd-debuginfo-19.0.0-1269.g633ab857.el9.x86 4.1 MB/s | 1.3 MB 00:00 2024-02-06T21:39:03.593 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:39:03.593 INFO:teuthology.orchestra.run.smithi120.stdout:Total 4.0 MB/s | 1.3 MB 00:00 2024-02-06T21:39:03.593 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-02-06T21:39:03.596 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-02-06T21:39:03.596 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-02-06T21:39:03.638 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-02-06T21:39:03.639 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-02-06T21:39:03.875 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-02-06T21:39:03.962 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:39:03.966 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:39:04.251 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_ 1/1 2024-02-06T21:39:04.251 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:04.251 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:39:04.251 INFO:teuthology.orchestra.run.smithi135.stdout: librados2-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:39:04.251 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:04.251 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:39:04.424 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:39:04.482 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librbd1-debuginfo 2024-02-06T21:39:04.689 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:39:04.689 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:04.689 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-02-06T21:39:04.689 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:39:04.689 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:04.689 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-02-06T21:39:04.861 DEBUG:teuthology.parallel:result is None 2024-02-06T21:39:05.051 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:09:26 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:39:05.210 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:39:05.210 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:05.210 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:39:05.211 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:05.211 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:39:05.211 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 52 M 2024-02-06T21:39:05.211 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:05.211 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:39:05.211 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:05.211 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:39:05.211 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:05.211 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 52 M 2024-02-06T21:39:05.211 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 308 M 2024-02-06T21:39:05.211 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:39:07.968 INFO:teuthology.orchestra.run.smithi135.stdout:librbd1-debuginfo-19.0.0-1269.g633ab857.el9.x86 19 MB/s | 52 MB 00:02 2024-02-06T21:39:07.968 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:39:07.969 INFO:teuthology.orchestra.run.smithi135.stdout:Total 19 MB/s | 52 MB 00:02 2024-02-06T21:39:07.969 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:39:07.973 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:39:07.973 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:39:08.323 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:39:08.324 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:39:14.355 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:39:14.450 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librbd1-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:39:16.814 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:39:17.099 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:39:17.099 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:17.099 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:39:17.099 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:39:17.099 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:17.100 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:39:17.309 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-cephfs-debuginfo 2024-02-06T21:39:17.878 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:09:38 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:39:18.033 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:39:18.034 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:18.034 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:39:18.034 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:18.034 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:39:18.034 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 615 k 2024-02-06T21:39:18.034 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:18.034 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:39:18.034 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:18.034 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:39:18.034 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:18.034 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 615 k 2024-02-06T21:39:18.034 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 2.1 M 2024-02-06T21:39:18.035 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:39:18.695 INFO:teuthology.orchestra.run.smithi135.stdout:python3-cephfs-debuginfo-19.0.0-1269.g633ab857. 931 kB/s | 615 kB 00:00 2024-02-06T21:39:18.696 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:39:18.696 INFO:teuthology.orchestra.run.smithi135.stdout:Total 929 kB/s | 615 kB 00:00 2024-02-06T21:39:18.696 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:39:18.700 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:39:18.700 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:39:18.737 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:39:18.737 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:39:18.904 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:39:19.002 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1269.g633ab857.el9 1/1 2024-02-06T21:39:19.454 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1269.g633ab857.el9 1/1 2024-02-06T21:39:19.730 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1269.g633ab857.el9 1/1 2024-02-06T21:39:19.731 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:19.731 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:39:19.731 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:39:19.731 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:19.731 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:39:19.916 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rados-debuginfo 2024-02-06T21:39:20.488 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:09:41 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:39:20.646 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:39:20.647 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:20.647 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:39:20.647 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:20.647 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:39:20.647 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 770 k 2024-02-06T21:39:20.647 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:20.647 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:39:20.647 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:20.647 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:39:20.647 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:20.647 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 770 k 2024-02-06T21:39:20.647 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 2.7 M 2024-02-06T21:39:20.648 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:39:20.948 INFO:teuthology.orchestra.run.smithi135.stdout:python3-rados-debuginfo-19.0.0-1269.g633ab857.e 2.5 MB/s | 770 kB 00:00 2024-02-06T21:39:20.949 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:39:20.949 INFO:teuthology.orchestra.run.smithi135.stdout:Total 2.5 MB/s | 770 kB 00:00 2024-02-06T21:39:20.949 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:39:20.952 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:39:20.952 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:39:20.990 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:39:20.990 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:39:21.169 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:39:21.272 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1269.g633ab857.el9. 1/1 2024-02-06T21:39:21.718 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1269.g633ab857.el9. 1/1 2024-02-06T21:39:21.996 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1269.g633ab857.el9. 1/1 2024-02-06T21:39:21.996 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:21.996 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:39:21.996 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:39:21.997 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:21.997 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:39:22.204 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rbd-debuginfo 2024-02-06T21:39:22.780 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:09:43 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:39:22.934 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:39:22.934 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:22.934 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:39:22.935 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:22.935 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:39:22.935 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 1.1 M 2024-02-06T21:39:22.935 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:22.935 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:39:22.935 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:22.935 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:39:22.935 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:22.935 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 1.1 M 2024-02-06T21:39:22.936 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 4.3 M 2024-02-06T21:39:22.936 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:39:23.294 INFO:teuthology.orchestra.run.smithi135.stdout:python3-rbd-debuginfo-19.0.0-1269.g633ab857.el9 3.1 MB/s | 1.1 MB 00:00 2024-02-06T21:39:23.294 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:39:23.294 INFO:teuthology.orchestra.run.smithi135.stdout:Total 3.1 MB/s | 1.1 MB 00:00 2024-02-06T21:39:23.294 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:39:23.298 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:39:23.298 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:39:23.338 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:39:23.338 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:39:23.560 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:39:23.684 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:39:24.127 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:39:24.404 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:39:24.404 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:24.404 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:39:24.404 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:39:24.404 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:24.404 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:39:24.597 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rgw-debuginfo 2024-02-06T21:39:25.170 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:09:46 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:39:25.325 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:39:25.326 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:25.326 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:39:25.326 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:25.326 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:39:25.326 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 337 k 2024-02-06T21:39:25.326 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:25.326 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:39:25.326 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:25.326 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:39:25.327 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:25.327 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 337 k 2024-02-06T21:39:25.327 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 1.0 M 2024-02-06T21:39:25.327 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:39:25.580 INFO:teuthology.orchestra.run.smithi135.stdout:python3-rgw-debuginfo-19.0.0-1269.g633ab857.el9 1.3 MB/s | 337 kB 00:00 2024-02-06T21:39:25.580 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:39:25.581 INFO:teuthology.orchestra.run.smithi135.stdout:Total 1.3 MB/s | 337 kB 00:00 2024-02-06T21:39:25.581 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:39:25.584 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:39:25.584 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:39:25.619 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:39:25.619 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:39:25.760 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:39:25.869 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:39:26.273 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:39:26.556 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x8 1/1 2024-02-06T21:39:26.557 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:26.557 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:39:26.557 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:39:26.557 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:26.557 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:39:26.744 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-fuse-debuginfo 2024-02-06T21:39:27.315 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:09:48 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:39:27.470 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:39:27.470 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:27.470 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:39:27.470 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:27.470 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:39:27.471 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 509 k 2024-02-06T21:39:27.471 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:27.471 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:39:27.471 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:27.471 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:39:27.471 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:27.471 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 509 k 2024-02-06T21:39:27.471 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 2.5 M 2024-02-06T21:39:27.472 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:39:27.866 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-fuse-debuginfo-19.0.0-1269.g633ab857.el9.x8 1.3 MB/s | 509 kB 00:00 2024-02-06T21:39:27.867 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:39:27.867 INFO:teuthology.orchestra.run.smithi135.stdout:Total 1.3 MB/s | 509 kB 00:00 2024-02-06T21:39:27.867 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:39:27.870 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:39:27.870 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:39:27.905 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:39:27.906 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:39:28.065 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:39:28.165 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:39:28.584 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:39:28.872 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_6 1/1 2024-02-06T21:39:28.872 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:28.872 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:39:28.872 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:39:28.872 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:28.872 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:39:29.083 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-mirror-debuginfo 2024-02-06T21:39:29.659 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:09:50 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:39:29.814 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:39:29.814 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:29.814 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:39:29.814 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:29.814 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:39:29.814 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 52 M 2024-02-06T21:39:29.814 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:29.814 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:39:29.815 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:29.815 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:39:29.815 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:29.815 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 52 M 2024-02-06T21:39:29.815 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 298 M 2024-02-06T21:39:29.816 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:39:32.352 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-mirror-debuginfo-19.0.0-1269.g633ab857.el9. 20 MB/s | 52 MB 00:02 2024-02-06T21:39:32.353 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:39:32.353 INFO:teuthology.orchestra.run.smithi135.stdout:Total 20 MB/s | 52 MB 00:02 2024-02-06T21:39:32.354 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:39:32.358 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:39:32.358 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:39:32.696 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:39:32.696 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:39:38.618 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:39:38.717 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1269.g633ab857.el9.x86 1/1 2024-02-06T21:39:41.010 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1269.g633ab857.el9.x86 1/1 2024-02-06T21:39:41.296 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1269.g633ab857.el9.x86 1/1 2024-02-06T21:39:41.296 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:41.296 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:39:41.296 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:39:41.296 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:41.296 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:39:41.492 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-nbd-debuginfo 2024-02-06T21:39:42.067 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:10:03 ago on Tue 06 Feb 2024 09:29:39 PM UTC. 2024-02-06T21:39:42.223 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-02-06T21:39:42.223 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:42.223 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-02-06T21:39:42.223 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:42.223 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-02-06T21:39:42.223 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1269.g633ab857.el9 ceph 1.3 M 2024-02-06T21:39:42.223 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:42.224 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-02-06T21:39:42.224 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-02-06T21:39:42.224 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-02-06T21:39:42.224 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:42.224 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 1.3 M 2024-02-06T21:39:42.224 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 5.5 M 2024-02-06T21:39:42.225 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-02-06T21:39:42.696 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-nbd-debuginfo-19.0.0-1269.g633ab857.el9.x86 2.7 MB/s | 1.3 MB 00:00 2024-02-06T21:39:42.697 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-02-06T21:39:42.697 INFO:teuthology.orchestra.run.smithi135.stdout:Total 2.6 MB/s | 1.3 MB 00:00 2024-02-06T21:39:42.697 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-02-06T21:39:42.700 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-02-06T21:39:42.701 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-02-06T21:39:42.742 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-02-06T21:39:42.742 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-02-06T21:39:42.982 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-02-06T21:39:43.087 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:39:43.576 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:39:43.864 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 1/1 2024-02-06T21:39:43.864 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:43.864 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-02-06T21:39:43.864 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd-debuginfo-2:19.0.0-1269.g633ab857.el9.x86_64 2024-02-06T21:39:43.864 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:43.864 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-02-06T21:39:44.074 DEBUG:teuthology.parallel:result is None 2024-02-06T21:39:44.075 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=633ab857b9926af935a3e6291c3e1d9251aca357 2024-02-06T21:39:44.292 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-06T21:39:44.316 INFO:teuthology.orchestra.run.smithi120.stdout:19.0.0-1269.g633ab857.el9 2024-02-06T21:39:44.317 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1269.g633ab857.el9 2024-02-06T21:39:44.317 INFO:teuthology.task.install:The correct ceph version 19.0.0-1269.g633ab857 is installed. 2024-02-06T21:39:44.319 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=633ab857b9926af935a3e6291c3e1d9251aca357 2024-02-06T21:39:44.532 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-06T21:39:44.553 INFO:teuthology.orchestra.run.smithi135.stdout:19.0.0-1269.g633ab857.el9 2024-02-06T21:39:44.553 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1269.g633ab857.el9 2024-02-06T21:39:44.554 INFO:teuthology.task.install:The correct ceph version 19.0.0-1269.g633ab857 is installed. 2024-02-06T21:39:44.555 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-06T21:39:44.556 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:39:44.556 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-06T21:39:44.601 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:39:44.601 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-06T21:39:44.628 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-06T21:39:44.629 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:39:44.629 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/daemon-helper 2024-02-06T21:39:44.673 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-06T21:39:44.742 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:39:44.742 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/daemon-helper 2024-02-06T21:39:44.769 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-06T21:39:44.836 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-06T21:39:44.836 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:39:44.836 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-06T21:39:44.864 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-06T21:39:44.932 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:39:44.932 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-06T21:39:44.959 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-06T21:39:45.025 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-06T21:39:45.025 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:39:45.025 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/stdin-killer 2024-02-06T21:39:45.054 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-06T21:39:45.120 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:39:45.121 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/stdin-killer 2024-02-06T21:39:45.147 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-06T21:39:45.214 INFO:teuthology.run_tasks:Running task ceph... 2024-02-06T21:39:45.299 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-02-06T21:39:45.299 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 777 /var/log/ceph 2024-02-06T21:39:45.301 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 777 /var/log/ceph 2024-02-06T21:39:45.327 INFO:tasks.ceph:Disabling ceph logrotate... 2024-02-06T21:39:45.327 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-06T21:39:45.369 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-06T21:39:45.395 INFO:tasks.ceph:Creating extra log directories... 2024-02-06T21:39:45.395 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-06T21:39:45.436 DEBUG:teuthology.orchestra.run.smithi135:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-06T21:39:45.472 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-02-06T21:39:45.472 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-02-06T21:39:45.472 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974', 'branch': 'wip-yuri4-testing-2024-02-05-0849', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7548974', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2024-02-05-0849'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '633ab857b9926af935a3e6291c3e1d9251aca357', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '633ab857b9926af935a3e6291c3e1d9251aca357'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2024-02-05-0849-no-mon-cluster-logging', 'sha1': '2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc'}}, 'owner': 'scheduled_lflores@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 4200, 'sha1': '633ab857b9926af935a3e6291c3e1d9251aca357', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2024-02-05-0849-no-mon-cluster-logging', 'suite_path': '/home/teuthworker/src/github.com_ljflores_ceph_2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ljflores/ceph.git', 'suite_sha1': '2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc', 'targets': {'smithi120.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBERq3tD9axCtMLTUR91xYMSSyspJePB3jBEHX7Vx31cW7gZaKDEvhcIP2WBNlHrWqXXnbvMeAgpeFcKGIFWh2wY=', 'smithi135.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE6IG63DZKKtm3VBcdAQFlU7KcQl66IeJPSg7EJNL9LVQtVfynSBOJoPR4umbHvTxWK70Dd61OqnVyNHcVGH45g='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'd9fdb2209e15b39d9f061fd85399f352ce0f0894', 'timestamp': '2024-02-06_20:55:47', 'tube': 'smithi', 'user': 'lflores', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-02-06T21:39:45.473 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-02-06T21:39:45.504 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-02-06T21:39:45.519 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m0777 -- /var/run/ceph 2024-02-06T21:39:45.559 DEBUG:teuthology.orchestra.run.smithi135:> sudo install -d -m0777 -- /var/run/ceph 2024-02-06T21:39:45.585 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:39:45.585 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2024-02-06T21:39:45.639 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-06T21:39:45.640 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_1 2024-02-06T21:39:45.694 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-06T21:39:45.694 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-06T21:39:45.694 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-06T21:39:45.694 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-06T21:39:45.694 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-02-06T21:39:45.694 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-02-06 21:31:16.294882023 +0000 2024-02-06T21:39:45.694 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-02-06 21:28:39.556137265 +0000 2024-02-06T21:39:45.694 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-02-06 21:28:39.556137265 +0000 2024-02-06T21:39:45.694 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-02-06 21:28:39.556137265 +0000 2024-02-06T21:39:45.695 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-06T21:39:45.760 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-02-06T21:39:45.760 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-02-06T21:39:45.760 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000202347 s, 2.5 MB/s 2024-02-06T21:39:45.761 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-06T21:39:45.817 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_2 2024-02-06T21:39:45.872 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-06T21:39:45.872 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-06T21:39:45.872 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-06T21:39:45.872 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-06T21:39:45.872 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-02-06T21:39:45.872 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-02-06 21:31:16.295882001 +0000 2024-02-06T21:39:45.872 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-02-06 21:28:39.891129730 +0000 2024-02-06T21:39:45.873 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-02-06 21:28:39.891129730 +0000 2024-02-06T21:39:45.873 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-02-06 21:28:39.891129730 +0000 2024-02-06T21:39:45.873 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-06T21:39:45.937 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-02-06T21:39:45.937 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-02-06T21:39:45.937 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000190731 s, 2.7 MB/s 2024-02-06T21:39:45.938 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-06T21:39:45.994 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_3 2024-02-06T21:39:46.048 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-06T21:39:46.048 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-06T21:39:46.048 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-06T21:39:46.048 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-06T21:39:46.048 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-02-06T21:39:46.048 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-02-06 21:31:16.295882001 +0000 2024-02-06T21:39:46.048 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-02-06 21:28:40.249121679 +0000 2024-02-06T21:39:46.049 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-02-06 21:28:40.249121679 +0000 2024-02-06T21:39:46.049 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-02-06 21:28:40.249121679 +0000 2024-02-06T21:39:46.049 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-06T21:39:46.114 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-02-06T21:39:46.114 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-02-06T21:39:46.114 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000211501 s, 2.4 MB/s 2024-02-06T21:39:46.116 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-06T21:39:46.172 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_4 2024-02-06T21:39:46.227 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-06T21:39:46.227 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-06T21:39:46.227 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-02-06T21:39:46.227 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-06T21:39:46.227 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-02-06T21:39:46.227 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-02-06 21:31:16.295882001 +0000 2024-02-06T21:39:46.227 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-02-06 21:28:40.604113694 +0000 2024-02-06T21:39:46.227 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-02-06 21:28:40.604113694 +0000 2024-02-06T21:39:46.227 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-02-06 21:28:40.604113694 +0000 2024-02-06T21:39:46.228 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-06T21:39:46.292 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-02-06T21:39:46.292 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-02-06T21:39:46.293 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000230636 s, 2.2 MB/s 2024-02-06T21:39:46.294 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-06T21:39:46.350 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-06T21:39:46.350 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:39:46.350 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/scratch_devs of=/dev/stdout 2024-02-06T21:39:46.366 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-06T21:39:46.366 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_1 2024-02-06T21:39:46.421 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-06T21:39:46.421 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-06T21:39:46.421 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-06T21:39:46.421 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-06T21:39:46.421 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-02-06T21:39:46.421 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-02-06 21:31:24.442922637 +0000 2024-02-06T21:39:46.422 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-02-06 21:28:39.550255425 +0000 2024-02-06T21:39:46.422 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-02-06 21:28:39.550255425 +0000 2024-02-06T21:39:46.422 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-02-06 21:28:39.550255425 +0000 2024-02-06T21:39:46.422 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-06T21:39:46.487 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-02-06T21:39:46.488 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-02-06T21:39:46.488 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.00018986 s, 2.7 MB/s 2024-02-06T21:39:46.489 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-06T21:39:46.545 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_2 2024-02-06T21:39:46.600 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-06T21:39:46.600 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-06T21:39:46.600 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-06T21:39:46.600 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-06T21:39:46.600 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-02-06T21:39:46.600 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-02-06 21:31:24.442922637 +0000 2024-02-06T21:39:46.600 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-02-06 21:28:39.923247524 +0000 2024-02-06T21:39:46.600 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-02-06 21:28:39.923247524 +0000 2024-02-06T21:39:46.600 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-02-06 21:28:39.923247524 +0000 2024-02-06T21:39:46.601 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-06T21:39:46.664 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-02-06T21:39:46.665 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-02-06T21:39:46.665 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000203154 s, 2.5 MB/s 2024-02-06T21:39:46.666 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-06T21:39:46.721 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_3 2024-02-06T21:39:46.775 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-06T21:39:46.775 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-06T21:39:46.775 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-06T21:39:46.775 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-06T21:39:46.776 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-02-06T21:39:46.776 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-02-06 21:31:24.443922616 +0000 2024-02-06T21:39:46.776 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-02-06 21:28:40.246240683 +0000 2024-02-06T21:39:46.776 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-02-06 21:28:40.246240683 +0000 2024-02-06T21:39:46.776 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-02-06 21:28:40.246240683 +0000 2024-02-06T21:39:46.776 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-06T21:39:46.841 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-02-06T21:39:46.841 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-02-06T21:39:46.841 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000207041 s, 2.5 MB/s 2024-02-06T21:39:46.842 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-06T21:39:46.898 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_4 2024-02-06T21:39:46.952 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-06T21:39:46.952 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-06T21:39:46.952 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-06T21:39:46.952 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-06T21:39:46.953 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-02-06T21:39:46.953 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-02-06 21:31:24.443922616 +0000 2024-02-06T21:39:46.953 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-02-06 21:28:40.576233693 +0000 2024-02-06T21:39:46.953 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-02-06 21:28:40.576233693 +0000 2024-02-06T21:39:46.953 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-02-06 21:28:40.576233693 +0000 2024-02-06T21:39:46.953 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-06T21:39:47.018 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-02-06T21:39:47.018 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-02-06T21:39:47.018 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000291584 s, 1.8 MB/s 2024-02-06T21:39:47.020 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-06T21:39:47.075 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-06T21:39:47.075 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi120.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi135.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-02-06T21:39:47.075 INFO:tasks.ceph:Generating config... 2024-02-06T21:39:47.076 INFO:tasks.ceph:[client] debug monc = 20 2024-02-06T21:39:47.076 INFO:tasks.ceph:[client] debug ms = 1 2024-02-06T21:39:47.077 INFO:tasks.ceph:[client] debug objecter = 20 2024-02-06T21:39:47.077 INFO:tasks.ceph:[client] debug rados = 20 2024-02-06T21:39:47.077 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-02-06T21:39:47.077 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-02-06T21:39:47.077 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-02-06T21:39:47.077 INFO:tasks.ceph:[global] ms type = async 2024-02-06T21:39:47.077 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-02-06T21:39:47.078 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-02-06T21:39:47.078 INFO:tasks.ceph:[mgr] debug ms = 1 2024-02-06T21:39:47.078 INFO:tasks.ceph:[mon] debug mon = 20 2024-02-06T21:39:47.078 INFO:tasks.ceph:[mon] debug ms = 1 2024-02-06T21:39:47.078 INFO:tasks.ceph:[mon] debug paxos = 20 2024-02-06T21:39:47.078 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-02-06T21:39:47.078 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-02-06T21:39:47.078 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-02-06T21:39:47.078 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-02-06T21:39:47.078 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-02-06T21:39:47.078 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-02-06T21:39:47.079 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-02-06T21:39:47.079 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-02-06T21:39:47.079 INFO:tasks.ceph:[osd] bdev async discard = True 2024-02-06T21:39:47.079 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-02-06T21:39:47.079 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-02-06T21:39:47.079 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-02-06T21:39:47.079 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-02-06T21:39:47.079 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-02-06T21:39:47.079 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-02-06T21:39:47.079 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-02-06T21:39:47.079 INFO:tasks.ceph:[osd] debug monc = 20 2024-02-06T21:39:47.079 INFO:tasks.ceph:[osd] debug ms = 1 2024-02-06T21:39:47.079 INFO:tasks.ceph:[osd] debug osd = 20 2024-02-06T21:39:47.079 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-02-06T21:39:47.079 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-02-06T21:39:47.080 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-02-06T21:39:47.080 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-02-06T21:39:47.080 INFO:tasks.ceph:[osd] osd class default list = * 2024-02-06T21:39:47.080 INFO:tasks.ceph:[osd] osd class load list = * 2024-02-06T21:39:47.080 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-02-06T21:39:47.080 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-02-06T21:39:47.080 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-02-06T21:39:47.080 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-02-06T21:39:47.080 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-02-06T21:39:47.080 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-02-06T21:39:47.080 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-02-06T21:39:47.080 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-02-06T21:39:47.080 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-02-06T21:39:47.081 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-02-06T21:39:47.081 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-02-06T21:39:47.081 INFO:tasks.ceph:Setting up mon.a... 2024-02-06T21:39:47.081 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-02-06T21:39:47.133 INFO:teuthology.orchestra.run.smithi120.stdout:creating /etc/ceph/ceph.keyring 2024-02-06T21:39:47.136 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-02-06T21:39:47.181 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-06T21:39:47.208 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.120'), ('mon.c', '[v2:172.21.15.120:3301,v1:172.21.15.120:6790]'), ('mon.b', '172.21.15.135')] 2024-02-06T21:39:47.208 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790],172.21.15.135', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-02-06T21:39:47.209 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:39:47.209 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-02-06T21:39:47.263 DEBUG:teuthology.orchestra.run.smithi120:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.120 --addv c '[v2:172.21.15.120:3301,v1:172.21.15.120:6790]' --add b 172.21.15.135 --print /home/ubuntu/cephtest/ceph.monmap 2024-02-06T21:39:47.348 INFO:teuthology.orchestra.run.smithi120.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-02-06T21:39:47.348 INFO:teuthology.orchestra.run.smithi120.stdout:monmaptool: generated fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:39:47.349 INFO:teuthology.orchestra.run.smithi120.stdout:setting min_mon_release = quincy 2024-02-06T21:39:47.349 INFO:teuthology.orchestra.run.smithi120.stdout:epoch 0 2024-02-06T21:39:47.349 INFO:teuthology.orchestra.run.smithi120.stdout:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:39:47.349 INFO:teuthology.orchestra.run.smithi120.stdout:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:39:47.349 INFO:teuthology.orchestra.run.smithi120.stdout:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:39:47.349 INFO:teuthology.orchestra.run.smithi120.stdout:min_mon_release 17 (quincy) 2024-02-06T21:39:47.349 INFO:teuthology.orchestra.run.smithi120.stdout:election_strategy: 1 2024-02-06T21:39:47.349 INFO:teuthology.orchestra.run.smithi120.stdout:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:39:47.349 INFO:teuthology.orchestra.run.smithi120.stdout:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:39:47.349 INFO:teuthology.orchestra.run.smithi120.stdout:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:39:47.349 INFO:teuthology.orchestra.run.smithi120.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-02-06T21:39:47.350 DEBUG:teuthology.orchestra.run.smithi120:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-02-06T21:39:47.365 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4e4791bf-0a02-4478-9517-2d2e893f4f7c... 2024-02-06T21:39:47.366 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-06T21:39:47.408 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-06T21:39:47.455 INFO:teuthology.orchestra.run.smithi120.stdout:[global] 2024-02-06T21:39:47.455 INFO:teuthology.orchestra.run.smithi120.stdout: chdir = "" 2024-02-06T21:39:47.455 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-06T21:39:47.455 INFO:teuthology.orchestra.run.smithi120.stdout: auth supported = cephx 2024-02-06T21:39:47.455 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.455 INFO:teuthology.orchestra.run.smithi120.stdout: filestore xattr use omap = true 2024-02-06T21:39:47.456 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.456 INFO:teuthology.orchestra.run.smithi120.stdout: mon clock drift allowed = 1.000 2024-02-06T21:39:47.456 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.456 INFO:teuthology.orchestra.run.smithi120.stdout: osd crush chooseleaf type = 0 2024-02-06T21:39:47.456 INFO:teuthology.orchestra.run.smithi120.stdout: auth debug = true 2024-02-06T21:39:47.456 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.456 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on old message = true 2024-02-06T21:39:47.456 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on bug = true 2024-02-06T21:39:47.456 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.456 INFO:teuthology.orchestra.run.smithi120.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-06T21:39:47.456 INFO:teuthology.orchestra.run.smithi120.stdout: mon pg warn max object skew = 0 2024-02-06T21:39:47.456 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.456 INFO:teuthology.orchestra.run.smithi120.stdout: # disable pg_autoscaler by default for new pools 2024-02-06T21:39:47.456 INFO:teuthology.orchestra.run.smithi120.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-06T21:39:47.457 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.457 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default size = 2 2024-02-06T21:39:47.457 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.457 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow primary affinity = true 2024-02-06T21:39:47.457 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow pg remap = true 2024-02-06T21:39:47.457 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on legacy crush tunables = false 2024-02-06T21:39:47.457 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on crush straw calc version zero = false 2024-02-06T21:39:47.457 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on no sortbitwise = false 2024-02-06T21:39:47.457 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on osd down out interval zero = false 2024-02-06T21:39:47.457 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on too few osds = false 2024-02-06T21:39:47.457 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-06T21:39:47.457 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-06T21:39:47.457 INFO:teuthology.orchestra.run.smithi120.stdout: mon_allow_pool_size_one = true 2024-02-06T21:39:47.457 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.458 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-06T21:39:47.458 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.458 INFO:teuthology.orchestra.run.smithi120.stdout: osd default data pool replay window = 5 2024-02-06T21:39:47.458 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.458 INFO:teuthology.orchestra.run.smithi120.stdout: mon allow pool delete = true 2024-02-06T21:39:47.458 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.458 INFO:teuthology.orchestra.run.smithi120.stdout: mon cluster log file level = debug 2024-02-06T21:39:47.458 INFO:teuthology.orchestra.run.smithi120.stdout: debug asserts on shutdown = true 2024-02-06T21:39:47.458 INFO:teuthology.orchestra.run.smithi120.stdout: mon health detail to clog = false 2024-02-06T21:39:47.458 INFO:teuthology.orchestra.run.smithi120.stdout: mon host = "172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790],172.21.15.135" 2024-02-06T21:39:47.458 INFO:teuthology.orchestra.run.smithi120.stdout: mon client directed command retry = 5 2024-02-06T21:39:47.458 INFO:teuthology.orchestra.run.smithi120.stdout: mon election default strategy = 1 2024-02-06T21:39:47.458 INFO:teuthology.orchestra.run.smithi120.stdout: ms inject socket failures = 5000 2024-02-06T21:39:47.458 INFO:teuthology.orchestra.run.smithi120.stdout: ms type = async 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout: osd heartbeat grace = 80 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout: fsid = 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout:[osd] 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout: osd journal size = 100 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub load threshold = 5.0 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub max interval = 600 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout: osd mclock profile = high_recovery_ops 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout: osd recover clone overlap = true 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout: osd recovery max chunk = 1048576 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug shutdown = true 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug op order = true 2024-02-06T21:39:47.459 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify stray on activate = true 2024-02-06T21:39:47.460 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug trim objects = true 2024-02-06T21:39:47.460 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.460 INFO:teuthology.orchestra.run.smithi120.stdout: osd open classes on start = true 2024-02-06T21:39:47.460 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug pg log writeout = true 2024-02-06T21:39:47.460 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.460 INFO:teuthology.orchestra.run.smithi120.stdout: osd deep scrub update digest min age = 30 2024-02-06T21:39:47.460 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.460 INFO:teuthology.orchestra.run.smithi120.stdout: osd map max advance = 10 2024-02-06T21:39:47.460 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.460 INFO:teuthology.orchestra.run.smithi120.stdout: journal zero on create = true 2024-02-06T21:39:47.460 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.460 INFO:teuthology.orchestra.run.smithi120.stdout: filestore ondisk finisher threads = 3 2024-02-06T21:39:47.460 INFO:teuthology.orchestra.run.smithi120.stdout: filestore apply finisher threads = 3 2024-02-06T21:39:47.460 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.461 INFO:teuthology.orchestra.run.smithi120.stdout: bdev debug aio = true 2024-02-06T21:39:47.461 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug misdirected ops = true 2024-02-06T21:39:47.461 INFO:teuthology.orchestra.run.smithi120.stdout: bdev async discard = True 2024-02-06T21:39:47.461 INFO:teuthology.orchestra.run.smithi120.stdout: bdev enable discard = True 2024-02-06T21:39:47.461 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore allocator = bitmap 2024-02-06T21:39:47.461 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore block size = 96636764160 2024-02-06T21:39:47.461 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore fsck on mount = True 2024-02-06T21:39:47.461 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore zero block detection = True 2024-02-06T21:39:47.461 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluefs = 1 2024-02-06T21:39:47.461 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluestore = 1 2024-02-06T21:39:47.461 INFO:teuthology.orchestra.run.smithi120.stdout: debug monc = 20 2024-02-06T21:39:47.461 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-02-06T21:39:47.461 INFO:teuthology.orchestra.run.smithi120.stdout: debug osd = 20 2024-02-06T21:39:47.461 INFO:teuthology.orchestra.run.smithi120.stdout: debug rocksdb = 10 2024-02-06T21:39:47.462 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd backfillfull_ratio = 0.85 2024-02-06T21:39:47.462 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd full ratio = 0.9 2024-02-06T21:39:47.462 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd nearfull ratio = 0.8 2024-02-06T21:39:47.462 INFO:teuthology.orchestra.run.smithi120.stdout: osd class default list = * 2024-02-06T21:39:47.462 INFO:teuthology.orchestra.run.smithi120.stdout: osd class load list = * 2024-02-06T21:39:47.462 INFO:teuthology.orchestra.run.smithi120.stdout: osd client watch timeout = 120 2024-02-06T21:39:47.462 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify cached snaps = True 2024-02-06T21:39:47.462 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify missing on start = True 2024-02-06T21:39:47.462 INFO:teuthology.orchestra.run.smithi120.stdout: osd failsafe full ratio = 0.95 2024-02-06T21:39:47.462 INFO:teuthology.orchestra.run.smithi120.stdout: osd fast shutdown = False 2024-02-06T21:39:47.462 INFO:teuthology.orchestra.run.smithi120.stdout: osd mclock override recovery settings = True 2024-02-06T21:39:47.462 INFO:teuthology.orchestra.run.smithi120.stdout: osd objectstore = bluestore 2024-02-06T21:39:47.462 INFO:teuthology.orchestra.run.smithi120.stdout: osd op queue = debug_random 2024-02-06T21:39:47.462 INFO:teuthology.orchestra.run.smithi120.stdout: osd op queue cut off = debug_random 2024-02-06T21:39:47.463 INFO:teuthology.orchestra.run.smithi120.stdout: osd shutdown pgref assert = True 2024-02-06T21:39:47.463 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.463 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr] 2024-02-06T21:39:47.463 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-02-06T21:39:47.463 INFO:teuthology.orchestra.run.smithi120.stdout: debug mgr = 20 2024-02-06T21:39:47.463 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-02-06T21:39:47.463 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-02-06T21:39:47.463 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-02-06T21:39:47.463 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-02-06T21:39:47.463 INFO:teuthology.orchestra.run.smithi120.stdout: mgr/telemetry/nag = false 2024-02-06T21:39:47.463 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.463 INFO:teuthology.orchestra.run.smithi120.stdout:[mon] 2024-02-06T21:39:47.463 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-02-06T21:39:47.463 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-02-06T21:39:47.463 INFO:teuthology.orchestra.run.smithi120.stdout: debug paxos = 20 2024-02-06T21:39:47.464 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-02-06T21:39:47.464 INFO:teuthology.orchestra.run.smithi120.stdout: mon data avail warn = 5 2024-02-06T21:39:47.464 INFO:teuthology.orchestra.run.smithi120.stdout: mon mgr mkfs grace = 240 2024-02-06T21:39:47.464 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-02-06T21:39:47.464 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd reporter subtree level = osd 2024-02-06T21:39:47.464 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd prime pg temp = true 2024-02-06T21:39:47.464 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-02-06T21:39:47.464 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.464 INFO:teuthology.orchestra.run.smithi120.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-06T21:39:47.464 INFO:teuthology.orchestra.run.smithi120.stdout: auth mon ticket ttl = 660 # 11m 2024-02-06T21:39:47.464 INFO:teuthology.orchestra.run.smithi120.stdout: auth service ticket ttl = 240 # 4m 2024-02-06T21:39:47.464 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.464 INFO:teuthology.orchestra.run.smithi120.stdout: # don't complain about insecure global_id in the test suite 2024-02-06T21:39:47.464 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-06T21:39:47.465 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-06T21:39:47.465 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.465 INFO:teuthology.orchestra.run.smithi120.stdout: # 1m isn't quite enough 2024-02-06T21:39:47.465 INFO:teuthology.orchestra.run.smithi120.stdout: mon_down_mkfs_grace = 2m 2024-02-06T21:39:47.465 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.465 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_filestore_osds = false 2024-02-06T21:39:47.465 INFO:teuthology.orchestra.run.smithi120.stdout: mon min osdmap epochs = 50 2024-02-06T21:39:47.465 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd crush smoke test = False 2024-02-06T21:39:47.465 INFO:teuthology.orchestra.run.smithi120.stdout: mon osdmap full prune interval = 2 2024-02-06T21:39:47.465 INFO:teuthology.orchestra.run.smithi120.stdout: mon osdmap full prune min = 15 2024-02-06T21:39:47.465 INFO:teuthology.orchestra.run.smithi120.stdout: mon osdmap full prune txsize = 2 2024-02-06T21:39:47.465 INFO:teuthology.orchestra.run.smithi120.stdout: mon scrub interval = 300 2024-02-06T21:39:47.465 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on pool no app = False 2024-02-06T21:39:47.465 INFO:teuthology.orchestra.run.smithi120.stdout: paxos service trim min = 10 2024-02-06T21:39:47.465 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:39:47.466 INFO:teuthology.orchestra.run.smithi120.stdout:[client] 2024-02-06T21:39:47.466 INFO:teuthology.orchestra.run.smithi120.stdout: rgw cache enabled = true 2024-02-06T21:39:47.466 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable ops log = true 2024-02-06T21:39:47.466 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable usage log = true 2024-02-06T21:39:47.466 INFO:teuthology.orchestra.run.smithi120.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-06T21:39:47.466 INFO:teuthology.orchestra.run.smithi120.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-06T21:39:47.466 INFO:teuthology.orchestra.run.smithi120.stdout: debug monc = 20 2024-02-06T21:39:47.466 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-02-06T21:39:47.466 INFO:teuthology.orchestra.run.smithi120.stdout: debug objecter = 20 2024-02-06T21:39:47.466 INFO:teuthology.orchestra.run.smithi120.stdout: debug rados = 20 2024-02-06T21:39:47.466 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.a] 2024-02-06T21:39:47.466 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.c] 2024-02-06T21:39:47.466 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.b] 2024-02-06T21:39:47.467 INFO:teuthology.orchestra.run.smithi135.stdout:[global] 2024-02-06T21:39:47.467 INFO:teuthology.orchestra.run.smithi135.stdout: chdir = "" 2024-02-06T21:39:47.467 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-06T21:39:47.467 INFO:teuthology.orchestra.run.smithi135.stdout: auth supported = cephx 2024-02-06T21:39:47.467 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.467 INFO:teuthology.orchestra.run.smithi135.stdout: filestore xattr use omap = true 2024-02-06T21:39:47.467 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.467 INFO:teuthology.orchestra.run.smithi135.stdout: mon clock drift allowed = 1.000 2024-02-06T21:39:47.467 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.467 INFO:teuthology.orchestra.run.smithi135.stdout: osd crush chooseleaf type = 0 2024-02-06T21:39:47.467 INFO:teuthology.orchestra.run.smithi135.stdout: auth debug = true 2024-02-06T21:39:47.468 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.468 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on old message = true 2024-02-06T21:39:47.468 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on bug = true 2024-02-06T21:39:47.468 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.468 INFO:teuthology.orchestra.run.smithi135.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-06T21:39:47.468 INFO:teuthology.orchestra.run.smithi135.stdout: mon pg warn max object skew = 0 2024-02-06T21:39:47.468 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.468 INFO:teuthology.orchestra.run.smithi135.stdout: # disable pg_autoscaler by default for new pools 2024-02-06T21:39:47.468 INFO:teuthology.orchestra.run.smithi135.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-06T21:39:47.468 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.468 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default size = 2 2024-02-06T21:39:47.468 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.468 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow primary affinity = true 2024-02-06T21:39:47.468 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow pg remap = true 2024-02-06T21:39:47.469 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on legacy crush tunables = false 2024-02-06T21:39:47.469 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on crush straw calc version zero = false 2024-02-06T21:39:47.469 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on no sortbitwise = false 2024-02-06T21:39:47.469 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on osd down out interval zero = false 2024-02-06T21:39:47.469 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on too few osds = false 2024-02-06T21:39:47.469 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-06T21:39:47.469 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-06T21:39:47.469 INFO:teuthology.orchestra.run.smithi135.stdout: mon_allow_pool_size_one = true 2024-02-06T21:39:47.469 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.469 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-06T21:39:47.469 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.469 INFO:teuthology.orchestra.run.smithi135.stdout: osd default data pool replay window = 5 2024-02-06T21:39:47.469 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.469 INFO:teuthology.orchestra.run.smithi135.stdout: mon allow pool delete = true 2024-02-06T21:39:47.470 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.470 INFO:teuthology.orchestra.run.smithi135.stdout: mon cluster log file level = debug 2024-02-06T21:39:47.470 INFO:teuthology.orchestra.run.smithi135.stdout: debug asserts on shutdown = true 2024-02-06T21:39:47.470 INFO:teuthology.orchestra.run.smithi135.stdout: mon health detail to clog = false 2024-02-06T21:39:47.470 INFO:teuthology.orchestra.run.smithi135.stdout: mon host = "172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790],172.21.15.135" 2024-02-06T21:39:47.470 INFO:teuthology.orchestra.run.smithi135.stdout: mon client directed command retry = 5 2024-02-06T21:39:47.470 INFO:teuthology.orchestra.run.smithi135.stdout: mon election default strategy = 1 2024-02-06T21:39:47.470 INFO:teuthology.orchestra.run.smithi135.stdout: ms inject socket failures = 5000 2024-02-06T21:39:47.470 INFO:teuthology.orchestra.run.smithi135.stdout: ms type = async 2024-02-06T21:39:47.470 INFO:teuthology.orchestra.run.smithi135.stdout: osd heartbeat grace = 80 2024-02-06T21:39:47.470 INFO:teuthology.orchestra.run.smithi135.stdout: fsid = 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:39:47.470 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.470 INFO:teuthology.orchestra.run.smithi135.stdout:[osd] 2024-02-06T21:39:47.470 INFO:teuthology.orchestra.run.smithi135.stdout: osd journal size = 100 2024-02-06T21:39:47.471 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.471 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub load threshold = 5.0 2024-02-06T21:39:47.471 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub max interval = 600 2024-02-06T21:39:47.471 INFO:teuthology.orchestra.run.smithi135.stdout: osd mclock profile = high_recovery_ops 2024-02-06T21:39:47.471 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.471 INFO:teuthology.orchestra.run.smithi135.stdout: osd recover clone overlap = true 2024-02-06T21:39:47.471 INFO:teuthology.orchestra.run.smithi135.stdout: osd recovery max chunk = 1048576 2024-02-06T21:39:47.471 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.471 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug shutdown = true 2024-02-06T21:39:47.471 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug op order = true 2024-02-06T21:39:47.471 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify stray on activate = true 2024-02-06T21:39:47.471 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug trim objects = true 2024-02-06T21:39:47.471 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.471 INFO:teuthology.orchestra.run.smithi135.stdout: osd open classes on start = true 2024-02-06T21:39:47.472 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug pg log writeout = true 2024-02-06T21:39:47.472 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.472 INFO:teuthology.orchestra.run.smithi135.stdout: osd deep scrub update digest min age = 30 2024-02-06T21:39:47.472 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.472 INFO:teuthology.orchestra.run.smithi135.stdout: osd map max advance = 10 2024-02-06T21:39:47.472 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.472 INFO:teuthology.orchestra.run.smithi135.stdout: journal zero on create = true 2024-02-06T21:39:47.472 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.472 INFO:teuthology.orchestra.run.smithi135.stdout: filestore ondisk finisher threads = 3 2024-02-06T21:39:47.472 INFO:teuthology.orchestra.run.smithi135.stdout: filestore apply finisher threads = 3 2024-02-06T21:39:47.472 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.472 INFO:teuthology.orchestra.run.smithi135.stdout: bdev debug aio = true 2024-02-06T21:39:47.472 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug misdirected ops = true 2024-02-06T21:39:47.472 INFO:teuthology.orchestra.run.smithi135.stdout: bdev async discard = True 2024-02-06T21:39:47.473 INFO:teuthology.orchestra.run.smithi135.stdout: bdev enable discard = True 2024-02-06T21:39:47.473 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore allocator = bitmap 2024-02-06T21:39:47.473 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore block size = 96636764160 2024-02-06T21:39:47.473 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore fsck on mount = True 2024-02-06T21:39:47.473 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore zero block detection = True 2024-02-06T21:39:47.473 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluefs = 1 2024-02-06T21:39:47.473 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluestore = 1 2024-02-06T21:39:47.473 INFO:teuthology.orchestra.run.smithi135.stdout: debug monc = 20 2024-02-06T21:39:47.473 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-02-06T21:39:47.473 INFO:teuthology.orchestra.run.smithi135.stdout: debug osd = 20 2024-02-06T21:39:47.473 INFO:teuthology.orchestra.run.smithi135.stdout: debug rocksdb = 10 2024-02-06T21:39:47.473 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd backfillfull_ratio = 0.85 2024-02-06T21:39:47.473 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd full ratio = 0.9 2024-02-06T21:39:47.473 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd nearfull ratio = 0.8 2024-02-06T21:39:47.474 INFO:teuthology.orchestra.run.smithi135.stdout: osd class default list = * 2024-02-06T21:39:47.474 INFO:teuthology.orchestra.run.smithi135.stdout: osd class load list = * 2024-02-06T21:39:47.474 INFO:teuthology.orchestra.run.smithi135.stdout: osd client watch timeout = 120 2024-02-06T21:39:47.474 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify cached snaps = True 2024-02-06T21:39:47.474 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify missing on start = True 2024-02-06T21:39:47.474 INFO:teuthology.orchestra.run.smithi135.stdout: osd failsafe full ratio = 0.95 2024-02-06T21:39:47.474 INFO:teuthology.orchestra.run.smithi135.stdout: osd fast shutdown = False 2024-02-06T21:39:47.474 INFO:teuthology.orchestra.run.smithi135.stdout: osd mclock override recovery settings = True 2024-02-06T21:39:47.474 INFO:teuthology.orchestra.run.smithi135.stdout: osd objectstore = bluestore 2024-02-06T21:39:47.474 INFO:teuthology.orchestra.run.smithi135.stdout: osd op queue = debug_random 2024-02-06T21:39:47.474 INFO:teuthology.orchestra.run.smithi135.stdout: osd op queue cut off = debug_random 2024-02-06T21:39:47.474 INFO:teuthology.orchestra.run.smithi135.stdout: osd shutdown pgref assert = True 2024-02-06T21:39:47.474 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.474 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr] 2024-02-06T21:39:47.474 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-02-06T21:39:47.475 INFO:teuthology.orchestra.run.smithi135.stdout: debug mgr = 20 2024-02-06T21:39:47.475 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-02-06T21:39:47.475 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-02-06T21:39:47.475 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-02-06T21:39:47.475 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-02-06T21:39:47.475 INFO:teuthology.orchestra.run.smithi135.stdout: mgr/telemetry/nag = false 2024-02-06T21:39:47.475 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.475 INFO:teuthology.orchestra.run.smithi135.stdout:[mon] 2024-02-06T21:39:47.475 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-02-06T21:39:47.475 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-02-06T21:39:47.475 INFO:teuthology.orchestra.run.smithi135.stdout: debug paxos = 20 2024-02-06T21:39:47.475 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-02-06T21:39:47.475 INFO:teuthology.orchestra.run.smithi135.stdout: mon data avail warn = 5 2024-02-06T21:39:47.475 INFO:teuthology.orchestra.run.smithi135.stdout: mon mgr mkfs grace = 240 2024-02-06T21:39:47.475 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-02-06T21:39:47.476 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd reporter subtree level = osd 2024-02-06T21:39:47.476 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd prime pg temp = true 2024-02-06T21:39:47.476 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-02-06T21:39:47.476 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.476 INFO:teuthology.orchestra.run.smithi135.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-06T21:39:47.476 INFO:teuthology.orchestra.run.smithi135.stdout: auth mon ticket ttl = 660 # 11m 2024-02-06T21:39:47.476 INFO:teuthology.orchestra.run.smithi135.stdout: auth service ticket ttl = 240 # 4m 2024-02-06T21:39:47.476 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.476 INFO:teuthology.orchestra.run.smithi135.stdout: # don't complain about insecure global_id in the test suite 2024-02-06T21:39:47.476 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-06T21:39:47.476 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-06T21:39:47.476 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.476 INFO:teuthology.orchestra.run.smithi135.stdout: # 1m isn't quite enough 2024-02-06T21:39:47.477 INFO:teuthology.orchestra.run.smithi135.stdout: mon_down_mkfs_grace = 2m 2024-02-06T21:39:47.477 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.477 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_filestore_osds = false 2024-02-06T21:39:47.477 INFO:teuthology.orchestra.run.smithi135.stdout: mon min osdmap epochs = 50 2024-02-06T21:39:47.477 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd crush smoke test = False 2024-02-06T21:39:47.477 INFO:teuthology.orchestra.run.smithi135.stdout: mon osdmap full prune interval = 2 2024-02-06T21:39:47.477 INFO:teuthology.orchestra.run.smithi135.stdout: mon osdmap full prune min = 15 2024-02-06T21:39:47.477 INFO:teuthology.orchestra.run.smithi135.stdout: mon osdmap full prune txsize = 2 2024-02-06T21:39:47.477 INFO:teuthology.orchestra.run.smithi135.stdout: mon scrub interval = 300 2024-02-06T21:39:47.477 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on pool no app = False 2024-02-06T21:39:47.477 INFO:teuthology.orchestra.run.smithi135.stdout: paxos service trim min = 10 2024-02-06T21:39:47.477 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-02-06T21:39:47.477 INFO:teuthology.orchestra.run.smithi135.stdout:[client] 2024-02-06T21:39:47.477 INFO:teuthology.orchestra.run.smithi135.stdout: rgw cache enabled = true 2024-02-06T21:39:47.477 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable ops log = true 2024-02-06T21:39:47.478 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable usage log = true 2024-02-06T21:39:47.478 INFO:teuthology.orchestra.run.smithi135.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-06T21:39:47.478 INFO:teuthology.orchestra.run.smithi135.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-06T21:39:47.478 INFO:teuthology.orchestra.run.smithi135.stdout: debug monc = 20 2024-02-06T21:39:47.478 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-02-06T21:39:47.478 INFO:teuthology.orchestra.run.smithi135.stdout: debug objecter = 20 2024-02-06T21:39:47.478 INFO:teuthology.orchestra.run.smithi135.stdout: debug rados = 20 2024-02-06T21:39:47.478 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.a] 2024-02-06T21:39:47.478 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.c] 2024-02-06T21:39:47.478 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.b] 2024-02-06T21:39:47.479 INFO:tasks.ceph:Creating admin key on mon.a... 2024-02-06T21:39:47.480 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-02-06T21:39:47.575 INFO:tasks.ceph:Copying monmap to all nodes... 2024-02-06T21:39:47.575 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:39:47.575 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-02-06T21:39:47.590 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:39:47.590 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-02-06T21:39:47.645 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi120.front.sepia.ceph.com 2024-02-06T21:39:47.645 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:39:47.645 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.keyring 2024-02-06T21:39:47.645 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-06T21:39:47.726 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:39:47.727 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-02-06T21:39:47.781 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi135.front.sepia.ceph.com 2024-02-06T21:39:47.781 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:39:47.781 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.keyring 2024-02-06T21:39:47.781 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-06T21:39:47.821 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:39:47.821 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-02-06T21:39:47.876 INFO:tasks.ceph:Setting up mon nodes... 2024-02-06T21:39:47.877 INFO:tasks.ceph:Setting up mgr nodes... 2024-02-06T21:39:47.877 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-02-06T21:39:47.930 INFO:teuthology.orchestra.run.smithi120.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-02-06T21:39:47.933 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-02-06T21:39:48.016 INFO:teuthology.orchestra.run.smithi135.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-02-06T21:39:48.019 INFO:tasks.ceph:Setting up mds nodes... 2024-02-06T21:39:48.019 INFO:tasks.ceph_client:Setting up client nodes... 2024-02-06T21:39:48.019 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-06T21:39:48.062 INFO:teuthology.orchestra.run.smithi120.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-02-06T21:39:48.077 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-06T21:39:48.119 INFO:teuthology.orchestra.run.smithi135.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-02-06T21:39:48.134 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-02-06T21:39:48.134 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi120.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi135.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-02-06T21:39:48.134 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-02-06T21:39:48.160 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-06T21:39:48.161 INFO:tasks.ceph:role: osd.0 2024-02-06T21:39:48.161 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi120.front.sepia.ceph.com 2024-02-06T21:39:48.161 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-06T21:39:48.234 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-06T21:39:48.234 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-06T21:39:48.234 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-06T21:39:48.234 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-06T21:39:48.234 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-06T21:39:48.234 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-02-06T21:39:48.234 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-06T21:39:48.234 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-06T21:39:48.234 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-06T21:39:48.234 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-06T21:39:48.747 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-02-06T21:39:48.749 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-02-06T21:39:48.749 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-02-06T21:39:48.891 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-02-06T21:39:48.924 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-02-06T21:39:48.992 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-06T21:39:48.992 INFO:tasks.ceph:role: osd.1 2024-02-06T21:39:48.992 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi120.front.sepia.ceph.com 2024-02-06T21:39:48.992 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-06T21:39:49.067 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-06T21:39:49.067 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-06T21:39:49.067 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-06T21:39:49.067 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-06T21:39:49.067 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-06T21:39:49.068 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-02-06T21:39:49.068 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-06T21:39:49.068 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-06T21:39:49.068 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-06T21:39:49.068 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-06T21:39:49.512 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-02-06T21:39:49.514 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-02-06T21:39:49.514 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-02-06T21:39:49.606 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-02-06T21:39:49.636 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-02-06T21:39:49.703 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-06T21:39:49.703 INFO:tasks.ceph:role: osd.2 2024-02-06T21:39:49.703 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi120.front.sepia.ceph.com 2024-02-06T21:39:49.703 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-06T21:39:49.777 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-06T21:39:49.777 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-06T21:39:49.777 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-06T21:39:49.777 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-06T21:39:49.777 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-06T21:39:49.778 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-02-06T21:39:49.778 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-06T21:39:49.778 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-06T21:39:49.778 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-06T21:39:49.778 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-06T21:39:50.278 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-02-06T21:39:50.280 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-02-06T21:39:50.280 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-02-06T21:39:50.420 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-02-06T21:39:50.452 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-02-06T21:39:50.519 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-06T21:39:50.519 INFO:tasks.ceph:role: osd.3 2024-02-06T21:39:50.519 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi120.front.sepia.ceph.com 2024-02-06T21:39:50.519 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-02-06T21:39:50.593 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-02-06T21:39:50.594 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-06T21:39:50.594 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-06T21:39:50.594 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-06T21:39:50.594 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-06T21:39:50.594 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-02-06T21:39:50.594 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-06T21:39:50.594 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-06T21:39:50.594 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-06T21:39:50.594 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-06T21:39:51.115 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-02-06T21:39:51.117 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-02-06T21:39:51.118 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-02-06T21:39:51.261 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-02-06T21:39:51.294 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-06T21:39:51.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:51.395+0000 7fbd104d1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-02-06T21:39:51.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:51.395+0000 7fbd104d1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-02-06T21:39:51.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:51.395+0000 7fbd104d1640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-06T21:39:51.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:51.395+0000 7fbd104d1640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-06T21:39:51.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:51.395+0000 7fbd104d1640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-06T21:39:51.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:51.396+0000 7fbd104d1640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-02-06T21:39:54.239 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-06T21:39:54.270 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-06T21:39:54.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:54.359+0000 7f0799a0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-02-06T21:39:54.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:54.360+0000 7f0799a0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-02-06T21:39:54.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:54.360+0000 7f0799a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-06T21:39:54.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:54.360+0000 7f0799a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-06T21:39:54.362 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:54.360+0000 7f0799a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-06T21:39:54.362 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:54.360+0000 7f0799a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-02-06T21:39:57.205 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-06T21:39:57.236 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-06T21:39:57.328 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:57.326+0000 7ff7eb225640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-02-06T21:39:57.328 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:57.327+0000 7ff7eb225640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-02-06T21:39:57.328 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:57.327+0000 7ff7eb225640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-06T21:39:57.328 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:57.327+0000 7ff7eb225640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-06T21:39:57.328 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:57.327+0000 7ff7eb225640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-06T21:39:57.329 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:39:57.327+0000 7ff7eb225640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-02-06T21:40:00.171 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-06T21:40:00.203 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-06T21:40:00.293 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:00.292+0000 7f32f5e06640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-02-06T21:40:00.293 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:00.293+0000 7f32f5e06640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-02-06T21:40:00.293 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:00.293+0000 7f32f5e06640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-02-06T21:40:00.294 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:00.293+0000 7f32f5e06640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-02-06T21:40:00.294 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:00.293+0000 7f32f5e06640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-02-06T21:40:00.294 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:00.293+0000 7f32f5e06640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-02-06T21:40:03.133 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-06T21:40:03.164 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-02-06T21:40:03.197 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-06T21:40:03.197 INFO:tasks.ceph:role: osd.4 2024-02-06T21:40:03.197 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi135.front.sepia.ceph.com 2024-02-06T21:40:03.197 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-06T21:40:03.276 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-06T21:40:03.277 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-06T21:40:03.277 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-06T21:40:03.277 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-06T21:40:03.277 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-06T21:40:03.277 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2024-02-06T21:40:03.277 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-06T21:40:03.277 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-06T21:40:03.277 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-06T21:40:03.277 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-06T21:40:03.765 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2024-02-06T21:40:03.767 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2024-02-06T21:40:03.767 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-02-06T21:40:03.876 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-02-06T21:40:03.915 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-02-06T21:40:03.984 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-06T21:40:03.985 INFO:tasks.ceph:role: osd.5 2024-02-06T21:40:03.985 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi135.front.sepia.ceph.com 2024-02-06T21:40:03.985 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-06T21:40:04.060 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-06T21:40:04.060 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-06T21:40:04.060 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-06T21:40:04.060 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-06T21:40:04.060 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-06T21:40:04.061 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2024-02-06T21:40:04.061 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-06T21:40:04.061 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-06T21:40:04.061 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-06T21:40:04.061 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-06T21:40:04.502 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2024-02-06T21:40:04.504 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2024-02-06T21:40:04.504 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-02-06T21:40:04.582 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-02-06T21:40:04.656 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-02-06T21:40:04.724 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-06T21:40:04.724 INFO:tasks.ceph:role: osd.6 2024-02-06T21:40:04.724 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi135.front.sepia.ceph.com 2024-02-06T21:40:04.724 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-06T21:40:04.799 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-06T21:40:04.799 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-06T21:40:04.799 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-06T21:40:04.800 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-06T21:40:04.800 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-06T21:40:04.800 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2024-02-06T21:40:04.800 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-06T21:40:04.800 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-06T21:40:04.800 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-06T21:40:04.800 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-06T21:40:05.284 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2024-02-06T21:40:05.286 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2024-02-06T21:40:05.286 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-02-06T21:40:05.382 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-02-06T21:40:05.414 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-02-06T21:40:05.481 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-06T21:40:05.481 INFO:tasks.ceph:role: osd.7 2024-02-06T21:40:05.481 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi135.front.sepia.ceph.com 2024-02-06T21:40:05.482 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-02-06T21:40:05.556 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-02-06T21:40:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-06T21:40:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-06T21:40:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-06T21:40:05.556 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-06T21:40:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2024-02-06T21:40:05.556 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-06T21:40:05.557 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-06T21:40:05.557 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-06T21:40:05.557 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-06T21:40:06.039 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2024-02-06T21:40:06.041 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2024-02-06T21:40:06.041 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-02-06T21:40:06.144 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-02-06T21:40:06.177 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-06T21:40:06.276 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:06.275+0000 7fa84180e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-02-06T21:40:06.281 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:06.280+0000 7fa84180e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-02-06T21:40:06.281 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:06.280+0000 7fa84180e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-02-06T21:40:06.281 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:06.280+0000 7fa84180e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-02-06T21:40:06.281 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:06.280+0000 7fa84180e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-02-06T21:40:06.282 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:06.282+0000 7fa84180e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-02-06T21:40:09.138 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-06T21:40:09.169 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-06T21:40:09.260 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:09.258+0000 7fcb1f938640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-02-06T21:40:09.260 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:09.258+0000 7fcb1f938640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-02-06T21:40:09.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:09.258+0000 7fcb1f938640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-02-06T21:40:09.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:09.258+0000 7fcb1f938640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-02-06T21:40:09.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:09.258+0000 7fcb1f938640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-02-06T21:40:09.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:09.259+0000 7fcb1f938640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-02-06T21:40:12.113 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-06T21:40:12.143 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-06T21:40:12.233 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:12.230+0000 7ff18d206640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-02-06T21:40:12.233 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:12.230+0000 7ff18d206640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-02-06T21:40:12.233 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:12.230+0000 7ff18d206640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-02-06T21:40:12.234 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:12.230+0000 7ff18d206640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-02-06T21:40:12.234 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:12.230+0000 7ff18d206640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-02-06T21:40:12.234 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:12.231+0000 7ff18d206640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-02-06T21:40:15.085 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-06T21:40:15.115 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-06T21:40:15.206 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:15.203+0000 7fdb1fed3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-02-06T21:40:15.206 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:15.203+0000 7fdb1fed3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-02-06T21:40:15.206 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:15.203+0000 7fdb1fed3640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-02-06T21:40:15.206 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:15.203+0000 7fdb1fed3640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-02-06T21:40:15.206 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:15.204+0000 7fdb1fed3640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-02-06T21:40:15.206 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:15.204+0000 7fdb1fed3640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-02-06T21:40:18.059 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-06T21:40:18.090 INFO:tasks.ceph:Reading keys from all nodes... 2024-02-06T21:40:18.090 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:40:18.090 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-02-06T21:40:18.122 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:40:18.122 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-02-06T21:40:18.193 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:40:18.193 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-02-06T21:40:18.263 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:40:18.263 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-02-06T21:40:18.332 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:40:18.332 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-02-06T21:40:18.400 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:40:18.400 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-02-06T21:40:18.432 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:40:18.432 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-02-06T21:40:18.503 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:40:18.504 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-02-06T21:40:18.573 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:40:18.574 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-02-06T21:40:18.642 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:40:18.642 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-02-06T21:40:18.711 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:40:18.711 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-02-06T21:40:18.728 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:40:18.728 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-02-06T21:40:18.767 INFO:tasks.ceph:Adding keys to all mons... 2024-02-06T21:40:18.767 DEBUG:teuthology.orchestra.run.smithi120:> sudo tee -a /etc/ceph/ceph.keyring 2024-02-06T21:40:18.770 DEBUG:teuthology.orchestra.run.smithi135:> sudo tee -a /etc/ceph/ceph.keyring 2024-02-06T21:40:18.811 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr.y] 2024-02-06T21:40:18.811 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAjp8JlqWl4NxAAiU0dgSGYxBckadz48ADSjA== 2024-02-06T21:40:18.811 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.0] 2024-02-06T21:40:18.812 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAnp8Jl2sKfFxAAap1Bf+IgtWV99oT5TLXq0w== 2024-02-06T21:40:18.812 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.1] 2024-02-06T21:40:18.812 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAqp8JloD+EFRAA5hHHYEQDImmMp3vQj/lQlw== 2024-02-06T21:40:18.812 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.2] 2024-02-06T21:40:18.812 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAtp8Jlm3t+ExAAyaP98B//Er+SxJxsCXyPog== 2024-02-06T21:40:18.812 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.3] 2024-02-06T21:40:18.812 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAwp8JlRgB6ERAAkOG+JzePf1ZixLpi5WJAHQ== 2024-02-06T21:40:18.812 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr.x] 2024-02-06T21:40:18.812 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAkp8JllRz7ABAAEs8Bu6NxTe/TjGFXkFMH+Q== 2024-02-06T21:40:18.812 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.4] 2024-02-06T21:40:18.812 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQA2p8Jlojt4EBAABs33qnHfRbBHPzJaBjwfeA== 2024-02-06T21:40:18.812 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.5] 2024-02-06T21:40:18.812 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQA5p8JlsOx9DxAAUCv0fhMZSTEC52f/fiRUNg== 2024-02-06T21:40:18.813 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.6] 2024-02-06T21:40:18.813 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQA8p8JlXcLUDRAAJNN+An2WzU5JEvaGSJRDDw== 2024-02-06T21:40:18.813 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.7] 2024-02-06T21:40:18.813 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQA/p8Jl9SM8DBAAJQDASPF2Auaa+n9rmn1fbQ== 2024-02-06T21:40:18.813 INFO:teuthology.orchestra.run.smithi120.stdout:[client.0] 2024-02-06T21:40:18.813 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAkp8JlfLu4AxAAHYDHsJnUqg99iukSubugJg== 2024-02-06T21:40:18.813 INFO:teuthology.orchestra.run.smithi120.stdout:[client.1] 2024-02-06T21:40:18.813 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAkp8JlKM4dBxAA2Sp+HG2B/hzgUgsJbidOxQ== 2024-02-06T21:40:18.834 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr.y] 2024-02-06T21:40:18.834 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQAjp8JlqWl4NxAAiU0dgSGYxBckadz48ADSjA== 2024-02-06T21:40:18.834 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.0] 2024-02-06T21:40:18.834 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQAnp8Jl2sKfFxAAap1Bf+IgtWV99oT5TLXq0w== 2024-02-06T21:40:18.834 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.1] 2024-02-06T21:40:18.834 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQAqp8JloD+EFRAA5hHHYEQDImmMp3vQj/lQlw== 2024-02-06T21:40:18.834 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.2] 2024-02-06T21:40:18.834 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQAtp8Jlm3t+ExAAyaP98B//Er+SxJxsCXyPog== 2024-02-06T21:40:18.834 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.3] 2024-02-06T21:40:18.834 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQAwp8JlRgB6ERAAkOG+JzePf1ZixLpi5WJAHQ== 2024-02-06T21:40:18.834 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr.x] 2024-02-06T21:40:18.834 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQAkp8JllRz7ABAAEs8Bu6NxTe/TjGFXkFMH+Q== 2024-02-06T21:40:18.834 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.4] 2024-02-06T21:40:18.835 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQA2p8Jlojt4EBAABs33qnHfRbBHPzJaBjwfeA== 2024-02-06T21:40:18.835 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.5] 2024-02-06T21:40:18.835 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQA5p8JlsOx9DxAAUCv0fhMZSTEC52f/fiRUNg== 2024-02-06T21:40:18.835 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.6] 2024-02-06T21:40:18.835 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQA8p8JlXcLUDRAAJNN+An2WzU5JEvaGSJRDDw== 2024-02-06T21:40:18.835 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.7] 2024-02-06T21:40:18.835 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQA/p8Jl9SM8DBAAJQDASPF2Auaa+n9rmn1fbQ== 2024-02-06T21:40:18.835 INFO:teuthology.orchestra.run.smithi135.stdout:[client.0] 2024-02-06T21:40:18.835 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQAkp8JlfLu4AxAAHYDHsJnUqg99iukSubugJg== 2024-02-06T21:40:18.835 INFO:teuthology.orchestra.run.smithi135.stdout:[client.1] 2024-02-06T21:40:18.835 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQAkp8JlKM4dBxAA2Sp+HG2B/hzgUgsJbidOxQ== 2024-02-06T21:40:18.836 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-06T21:40:18.857 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-06T21:40:18.929 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:18.931 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:18.982 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:18.984 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:19.035 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:19.037 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:19.090 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:19.092 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:19.144 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-06T21:40:19.146 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-06T21:40:19.197 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:19.199 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:19.250 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:19.252 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:19.307 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:19.309 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:19.361 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:19.362 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-06T21:40:19.414 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-06T21:40:19.416 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-06T21:40:19.471 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-06T21:40:19.473 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-06T21:40:19.526 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-02-06T21:40:19.527 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-02-06T21:40:19.553 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-06T21:40:19.649 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-02-06T21:40:19.676 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-02-06T21:40:19.743 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-06T21:40:19.840 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-02-06T21:40:19.867 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-02-06T21:40:19.894 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-06T21:40:19.992 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-02-06T21:40:20.018 DEBUG:teuthology.orchestra.run.smithi120:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-02-06T21:40:20.021 DEBUG:teuthology.orchestra.run.smithi135:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-02-06T21:40:20.076 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-02-06T21:40:20.076 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-02-06T21:40:20.076 INFO:tasks.ceph.mon.a:Restarting daemon 2024-02-06T21:40:20.076 DEBUG:teuthology.orchestra.run.smithi120:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-02-06T21:40:20.078 INFO:tasks.ceph.mon.a:Started 2024-02-06T21:40:20.079 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-02-06T21:40:20.079 INFO:tasks.ceph.mon.c:Restarting daemon 2024-02-06T21:40:20.079 DEBUG:teuthology.orchestra.run.smithi120:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-02-06T21:40:20.081 INFO:tasks.ceph.mon.c:Started 2024-02-06T21:40:20.081 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-02-06T21:40:20.081 INFO:tasks.ceph.mon.b:Restarting daemon 2024-02-06T21:40:20.081 DEBUG:teuthology.orchestra.run.smithi135:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-02-06T21:40:20.118 INFO:tasks.ceph.mon.b:Started 2024-02-06T21:40:20.118 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-02-06T21:40:20.119 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-02-06T21:40:20.119 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-02-06T21:40:20.121 INFO:tasks.ceph.mgr.y:Started 2024-02-06T21:40:20.121 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-02-06T21:40:20.121 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-02-06T21:40:20.123 INFO:tasks.ceph.mgr.x:Started 2024-02-06T21:40:20.123 INFO:tasks.ceph:Setting crush tunables to default 2024-02-06T21:40:20.124 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd crush tunables default 2024-02-06T21:40:20.224 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.223+0000 7f247338d640 10 monclient: get_monmap_and_config 2024-02-06T21:40:20.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.223+0000 7f247338d640 10 monclient: build_initial_monmap 2024-02-06T21:40:20.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.223+0000 7f247338d640 10 monclient: monmap: 2024-02-06T21:40:20.225 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:40:20.225 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:20.225 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:40:20.224611+0000 2024-02-06T21:40:20.225 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:40:20.224611+0000 2024-02-06T21:40:20.225 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:20.225 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:20.225 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:40:20.225 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:40:20.226 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:40:20.226 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:40:20.226 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:40:20.226 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:40:20.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.223+0000 7f247338d640 1 Processor -- start 2024-02-06T21:40:20.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 1 -- start start 2024-02-06T21:40:20.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 10 monclient: init 2024-02-06T21:40:20.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:20.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-02-06T21:40:20.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f246c0b54b0 0x7f246c0a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:20.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 10 monclient(hunting): picked mon.noname-b con 0x7f246c0b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:20.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f246c13f830 addr v1:172.21.15.135:6789/0 2024-02-06T21:40:20.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f246c142680 0x7f246c142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:20.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 10 monclient(hunting): picked mon.noname-c con 0x7f246c142680 addr v2:172.21.15.135:3300/0 2024-02-06T21:40:20.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246c0ab720 con 0x7f246c13f830 2024-02-06T21:40:20.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:20.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246c0ab520 con 0x7f246c142680 2024-02-06T21:40:20.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:20.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246c0b5890 con 0x7f246c0b54b0 2024-02-06T21:40:20.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:20.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247338d640 10 monclient(hunting): authenticate will time out at 1250.562323s 2024-02-06T21:40:20.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f2471129640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f246c0b54b0 msgr2=0x7f246c0a9860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-02-06T21:40:20.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f2471129640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f246c0b54b0 0x7f246c0a9860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-06T21:40:20.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f2470928640 1 -- >> v1:172.21.15.135:6789/0 conn(0x7f246c13f830 legacy=0x7f246c13fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.135:6789/0 2024-02-06T21:40:20.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247192a640 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f246c142680 msgr2=0x7f246c142af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2024-02-06T21:40:20.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.224+0000 7f247192a640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f246c142680 0x7f246c142af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-06T21:40:20.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.425+0000 7f2471129640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f246c0b54b0 msgr2=0x7f246c0a9860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-02-06T21:40:20.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.425+0000 7f2471129640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f246c0b54b0 0x7f246c0a9860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-06T21:40:20.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.425+0000 7f2470928640 1 -- >> v1:172.21.15.135:6789/0 conn(0x7f246c13f830 legacy=0x7f246c13fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.135:6789/0 2024-02-06T21:40:20.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.425+0000 7f247192a640 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f246c142680 msgr2=0x7f246c142af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2024-02-06T21:40:20.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.425+0000 7f247192a640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f246c142680 0x7f246c142af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-06T21:40:20.827 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.826+0000 7f2471129640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f246c0b54b0 msgr2=0x7f246c0a9860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-02-06T21:40:20.828 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.826+0000 7f2471129640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f246c0b54b0 0x7f246c0a9860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-06T21:40:20.828 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.826+0000 7f247192a640 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f246c142680 msgr2=0x7f246c142af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2024-02-06T21:40:20.828 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.826+0000 7f2470928640 1 -- >> v1:172.21.15.135:6789/0 conn(0x7f246c13f830 legacy=0x7f246c13fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.135:6789/0 2024-02-06T21:40:20.828 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:20.826+0000 7f247192a640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f246c142680 0x7f246c142af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-06T21:40:21.628 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:21.627+0000 7f2471129640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f246c0b54b0 msgr2=0x7f246c0a9860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-02-06T21:40:21.629 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:21.627+0000 7f2471129640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f246c0b54b0 0x7f246c0a9860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-06T21:40:21.629 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:21.627+0000 7f2470928640 1 -- >> v1:172.21.15.135:6789/0 conn(0x7f246c13f830 legacy=0x7f246c13fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.135:6789/0 2024-02-06T21:40:21.629 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:21.627+0000 7f247192a640 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f246c142680 msgr2=0x7f246c142af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2024-02-06T21:40:21.629 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:21.627+0000 7f247192a640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f246c142680 0x7f246c142af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-06T21:40:23.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 10 monclient(hunting): tick 2024-02-06T21:40:23.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 10 monclient(hunting): _check_auth_tickets 2024-02-06T21:40:23.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 1 monclient(hunting): continuing hunt 2024-02-06T21:40:23.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 10 monclient(hunting): _reopen_session rank -1 2024-02-06T21:40:23.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f246c0b54b0 msgr2=0x7f246c0a9860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-06T21:40:23.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f246c0b54b0 0x7f246c0a9860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:23.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f246c142680 msgr2=0x7f246c142af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-06T21:40:23.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f246c142680 0x7f246c142af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:23.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 1 -- >> v1:172.21.15.135:6789/0 conn(0x7f246c13f830 legacy=0x7f246c13fc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-06T21:40:23.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 10 monclient: _add_conns ranks=[0,3] 2024-02-06T21:40:23.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24500036f0 addr v1:172.21.15.120:6789/0 2024-02-06T21:40:23.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f24500073a0 0x7f2450009860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:23.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f24500073a0 addr v2:172.21.15.120:3300/0 2024-02-06T21:40:23.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246c0b4800 con 0x7f24500036f0 2024-02-06T21:40:23.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:23.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246c0ab520 con 0x7f24500073a0 2024-02-06T21:40:23.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2462ffd640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:23.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2471129640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f24500036f0 legacy=0x7f2450005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-02-06T21:40:23.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2470928640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f24500073a0 msgr2=0x7f2450009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-02-06T21:40:23.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.224+0000 7f2470928640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f24500073a0 0x7f2450009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-06T21:40:23.426 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.425+0000 7f2471129640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f24500036f0 legacy=0x7f2450005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-02-06T21:40:23.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.425+0000 7f2470928640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f24500073a0 msgr2=0x7f2450009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-02-06T21:40:23.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.425+0000 7f2470928640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f24500073a0 0x7f2450009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-06T21:40:23.827 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.826+0000 7f2471129640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f24500036f0 legacy=0x7f2450005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-02-06T21:40:23.827 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.826+0000 7f2470928640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f24500073a0 msgr2=0x7f2450009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-02-06T21:40:23.827 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:23.826+0000 7f2470928640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f24500073a0 0x7f2450009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-06T21:40:24.628 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:24.627+0000 7f2471129640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f24500036f0 legacy=0x7f2450005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-02-06T21:40:24.629 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:24.627+0000 7f2470928640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f24500073a0 msgr2=0x7f2450009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-02-06T21:40:24.629 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:24.627+0000 7f2470928640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f24500073a0 0x7f2450009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-06T21:40:26.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 10 monclient(hunting): tick 2024-02-06T21:40:26.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 10 monclient(hunting): _check_auth_tickets 2024-02-06T21:40:26.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 1 monclient(hunting): continuing hunt 2024-02-06T21:40:26.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 10 monclient(hunting): _reopen_session rank -1 2024-02-06T21:40:26.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f24500073a0 msgr2=0x7f2450009860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-06T21:40:26.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f24500073a0 0x7f2450009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:26.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f24500036f0 legacy=0x7f2450005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-06T21:40:26.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f247192a640 1 -- reap_dead start 2024-02-06T21:40:26.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-02-06T21:40:26.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 0x7f2450009860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:26.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f24500062a0 addr v2:172.21.15.135:3300/0 2024-02-06T21:40:26.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2450006720 0x7f2450005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:26.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f2450006720 addr v2:172.21.15.120:3300/0 2024-02-06T21:40:26.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2450006b00 0x7f2450009ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:26.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f2450006b00 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:26.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:26.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246c0ab520 con 0x7f2450006720 2024-02-06T21:40:26.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:26.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246c0b5890 con 0x7f24500062a0 2024-02-06T21:40:26.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:26.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f245000a420 con 0x7f2450006b00 2024-02-06T21:40:26.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2462ffd640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:26.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2470928640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f2450006720 msgr2=0x7f2450005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-02-06T21:40:26.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2470928640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2450006720 0x7f2450005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-06T21:40:26.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2471129640 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 msgr2=0x7f2450009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2024-02-06T21:40:26.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f247192a640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2450006b00 msgr2=0x7f2450009ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-02-06T21:40:26.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f2471129640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 0x7f2450009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-06T21:40:26.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.225+0000 7f247192a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2450006b00 0x7f2450009ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-06T21:40:26.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.426+0000 7f2470928640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f2450006720 msgr2=0x7f2450005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-02-06T21:40:26.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.426+0000 7f247192a640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2450006b00 msgr2=0x7f2450009ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-02-06T21:40:26.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.426+0000 7f2470928640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2450006720 0x7f2450005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-06T21:40:26.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.426+0000 7f247192a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2450006b00 0x7f2450009ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-06T21:40:26.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.426+0000 7f2471129640 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 msgr2=0x7f2450009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2024-02-06T21:40:26.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.426+0000 7f2471129640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 0x7f2450009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-06T21:40:26.828 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.827+0000 7f2470928640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f2450006720 msgr2=0x7f2450005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-02-06T21:40:26.828 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.827+0000 7f247192a640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2450006b00 msgr2=0x7f2450009ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-02-06T21:40:26.829 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.827+0000 7f2470928640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2450006720 0x7f2450005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-06T21:40:26.829 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.827+0000 7f247192a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2450006b00 0x7f2450009ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-06T21:40:26.829 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.827+0000 7f2471129640 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 msgr2=0x7f2450009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2024-02-06T21:40:26.829 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:26.827+0000 7f2471129640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 0x7f2450009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-06T21:40:27.144 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:00:06.908 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:40:27.419 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:00:07.234 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:40:27.419 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:00:07.234 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:40:27.453 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:00:07.217 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:40:27.629 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:27.628+0000 7f247192a640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2450006b00 msgr2=0x7f2450009ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-02-06T21:40:27.629 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:27.628+0000 7f2470928640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f2450006720 msgr2=0x7f2450005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-02-06T21:40:27.629 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:27.628+0000 7f247192a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2450006b00 0x7f2450009ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-06T21:40:27.630 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:27.628+0000 7f2470928640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2450006720 0x7f2450005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-06T21:40:27.630 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:27.628+0000 7f2471129640 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 msgr2=0x7f2450009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2024-02-06T21:40:27.630 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:27.628+0000 7f2471129640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 0x7f2450009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-06T21:40:27.730 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:00:07.545 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:40:27.731 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:00:07.546 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:40:29.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 10 monclient(hunting): tick 2024-02-06T21:40:29.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 10 monclient(hunting): _check_auth_tickets 2024-02-06T21:40:29.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 1 monclient(hunting): continuing hunt 2024-02-06T21:40:29.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 10 monclient(hunting): _reopen_session rank -1 2024-02-06T21:40:29.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2450006b00 msgr2=0x7f2450009ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-06T21:40:29.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2450006b00 0x7f2450009ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:29.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 msgr2=0x7f2450009860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-06T21:40:29.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 0x7f2450009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:29.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f2450006720 msgr2=0x7f2450005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-06T21:40:29.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2450006720 0x7f2450005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:29.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 10 monclient: _add_conns ranks=[1,0] 2024-02-06T21:40:29.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f245000a6a0 addr v1:172.21.15.135:6789/0 2024-02-06T21:40:29.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f245000aab0 addr v1:172.21.15.120:6789/0 2024-02-06T21:40:29.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246c0ab720 con 0x7f245000aab0 2024-02-06T21:40:29.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246c0b4800 con 0x7f245000a6a0 2024-02-06T21:40:29.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.225+0000 7f2462ffd640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:29.298 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.296+0000 7f2471129640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f245000a6a0 0x7f245000f060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.120:52836/0 (socket says 172.21.15.120:52836) 2024-02-06T21:40:29.298 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:29.296+0000 7f2471129640 1 -- 172.21.15.120:0/3254704083 learned_addr learned my addr 172.21.15.120:0/3254704083 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:40:32.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 10 monclient(hunting): tick 2024-02-06T21:40:32.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 10 monclient(hunting): _check_auth_tickets 2024-02-06T21:40:32.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 1 monclient(hunting): continuing hunt 2024-02-06T21:40:32.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 10 monclient(hunting): _reopen_session rank -1 2024-02-06T21:40:32.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 1 -- 172.21.15.120:0/3254704083 >> v1:172.21.15.135:6789/0 conn(0x7f245000a6a0 legacy=0x7f245000f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 1 -- 172.21.15.120:0/3254704083 >> v1:172.21.15.120:6789/0 conn(0x7f245000aab0 legacy=0x7f2450012790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-02-06T21:40:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f247192a640 1 -- 172.21.15.120:0/3254704083 reap_dead start 2024-02-06T21:40:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2450013f30 addr v1:172.21.15.135:6789/0 2024-02-06T21:40:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 1 --2- 172.21.15.120:0/3254704083 >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 0x7f2450009860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f24500062a0 addr v2:172.21.15.135:3300/0 2024-02-06T21:40:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 1 --2- 172.21.15.120:0/3254704083 >> v2:172.21.15.120:3300/0 conn(0x7f2450009ee0 0x7f2450005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f2450009ee0 addr v2:172.21.15.120:3300/0 2024-02-06T21:40:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 1 -- 172.21.15.120:0/3254704083 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2450015fa0 con 0x7f2450013f30 2024-02-06T21:40:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 1 -- 172.21.15.120:0/3254704083 --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246c0ab520 con 0x7f2450009ee0 2024-02-06T21:40:32.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:32.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 1 -- 172.21.15.120:0/3254704083 --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246c0b5890 con 0x7f24500062a0 2024-02-06T21:40:32.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.225+0000 7f2462ffd640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:32.265 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.263+0000 7f2470928640 1 --2- 172.21.15.120:0/3254704083 >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 0x7f2450009860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:32.265 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.264+0000 7f2470928640 10 monclient(hunting): get_auth_request con 0x7f24500062a0 auth_method 0 2024-02-06T21:40:32.265 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.264+0000 7f2470928640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:32.265 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.264+0000 7f2470928640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:32.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.264+0000 7f2470928640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:32.333 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.332+0000 7f247192a640 1 --2- 172.21.15.120:0/3254704083 >> v2:172.21.15.120:3300/0 conn(0x7f2450009ee0 0x7f2450005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:32.338 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.337+0000 7f247192a640 10 monclient(hunting): get_auth_request con 0x7f2450009ee0 auth_method 0 2024-02-06T21:40:32.338 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.337+0000 7f247192a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:32.338 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.337+0000 7f247192a640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:32.338 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.337+0000 7f247192a640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:32.347 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.344+0000 7f247192a640 1 -- 172.21.15.120:0/3254704083 >> v2:172.21.15.120:3300/0 conn(0x7f2450009ee0 msgr2=0x7f2450005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-02-06T21:40:32.347 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.344+0000 7f247192a640 1 -- 172.21.15.120:0/3254704083 >> v2:172.21.15.120:3300/0 conn(0x7f2450009ee0 msgr2=0x7f2450005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-06T21:40:32.347 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.344+0000 7f247192a640 1 --2- 172.21.15.120:0/3254704083 >> v2:172.21.15.120:3300/0 conn(0x7f2450009ee0 0x7f2450005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-06T21:40:32.348 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.344+0000 7f247192a640 1 --2- 172.21.15.120:0/3254704083 >> v2:172.21.15.120:3300/0 conn(0x7f2450009ee0 0x7f2450005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-06T21:40:32.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.430+0000 7f2470928640 1 -- 172.21.15.120:0/3254704083 >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 msgr2=0x7f2450009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-02-06T21:40:32.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.430+0000 7f2470928640 1 -- 172.21.15.120:0/3254704083 >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 msgr2=0x7f2450009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-06T21:40:32.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.430+0000 7f2470928640 1 --2- 172.21.15.120:0/3254704083 >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 0x7f2450009860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-06T21:40:32.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.430+0000 7f2470928640 1 --2- 172.21.15.120:0/3254704083 >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 0x7f2450009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-06T21:40:32.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.576+0000 7f247192a640 1 --2- 172.21.15.120:0/3254704083 >> v2:172.21.15.120:3300/0 conn(0x7f2450009ee0 0x7f2450005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:32.580 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.579+0000 7f247192a640 10 monclient(hunting): get_auth_request con 0x7f2450009ee0 auth_method 0 2024-02-06T21:40:32.580 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.579+0000 7f247192a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:32.580 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.579+0000 7f247192a640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:32.580 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.579+0000 7f247192a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-02-06T21:40:32.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.602+0000 7f2463fff640 1 -- 172.21.15.120:0/3254704083 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640083103 0 0) 0x7f2450015fa0 con 0x7f2450013f30 2024-02-06T21:40:32.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.602+0000 7f2463fff640 10 monclient(hunting): _negotiate 2024-02-06T21:40:32.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.602+0000 7f2463fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:32.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.602+0000 7f2463fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:32.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.602+0000 7f2463fff640 10 monclient(hunting): my global_id is 4102 2024-02-06T21:40:32.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.602+0000 7f2463fff640 1 -- 172.21.15.120:0/3254704083 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2458003520 con 0x7f2450013f30 2024-02-06T21:40:32.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.631+0000 7f247192a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:32.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.631+0000 7f247192a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:32.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.631+0000 7f247192a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.685+0000 7f2470928640 1 --2- 172.21.15.120:0/3254704083 >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 0x7f2450009860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.686+0000 7f2470928640 10 monclient(hunting): get_auth_request con 0x7f24500062a0 auth_method 0 2024-02-06T21:40:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.686+0000 7f2470928640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.686+0000 7f2470928640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.686+0000 7f2470928640 10 monclient(hunting): _init_auth already have auth, reseting 2024-02-06T21:40:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.687+0000 7f247192a640 10 monclient(hunting): handle_auth_done global_id 4104 payload 274 2024-02-06T21:40:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.687+0000 7f247192a640 1 -- 172.21.15.120:0/3254704083 >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 msgr2=0x7f2450009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.687+0000 7f247192a640 1 --2- 172.21.15.120:0/3254704083 >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 0x7f2450009860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.687+0000 7f247192a640 1 -- 172.21.15.120:0/3254704083 >> v1:172.21.15.135:6789/0 conn(0x7f2450013f30 legacy=0x7f2450014370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.687+0000 7f247192a640 10 monclient: _finish_hunting 0 2024-02-06T21:40:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.687+0000 7f247192a640 1 monclient: found mon.noname-a 2024-02-06T21:40:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.687+0000 7f247192a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.687+0000 7f247192a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:40:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.687+0000 7f247192a640 1 -- 172.21.15.120:0/3254704083 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2450014bf0 con 0x7f2450009ee0 2024-02-06T21:40:32.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.715+0000 7f247192a640 1 --2- 172.21.15.120:0/3254704083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2450009ee0 0x7f2450005b70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f246c0aa6a0 tx=0x7f246402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7ed005f71df0d6 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:32.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f2463fff640 1 -- 172.21.15.120:0/3254704083 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464033600 con 0x7f2450009ee0 2024-02-06T21:40:32.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f2463fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:32.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f2463fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:40:32.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f2463fff640 10 monclient: dump: 2024-02-06T21:40:32.773 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:40:32.773 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:32.773 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:32.773 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:32.773 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:32.773 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:32.773 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:32.773 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:32.773 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:32.773 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:40:32.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f2463fff640 10 monclient: _finish_auth 0 2024-02-06T21:40:32.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f2463fff640 10 monclient: _check_auth_tickets 2024-02-06T21:40:32.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f2463fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:32.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f2463fff640 1 -- 172.21.15.120:0/3254704083 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246402f050 con 0x7f2450009ee0 2024-02-06T21:40:32.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f2463fff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:32.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f2463fff640 1 -- 172.21.15.120:0/3254704083 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24640339a0 con 0x7f2450009ee0 2024-02-06T21:40:32.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f247338d640 5 monclient: authenticate success, global_id 4104 2024-02-06T21:40:32.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f247338d640 10 monclient: get_monmap_and_config success 2024-02-06T21:40:32.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f247338d640 4 set_mon_vals no callback set 2024-02-06T21:40:32.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f2463fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:32.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f2463fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:40:32.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.771+0000 7f2463fff640 10 monclient: dump: 2024-02-06T21:40:32.774 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:40:32.775 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:32.775 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:32.775 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:32.775 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:32.775 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:32.775 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:32.775 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:32.775 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:32.775 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:40:32.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.772+0000 7f247338d640 10 monclient: shutdown 2024-02-06T21:40:32.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.772+0000 7f247338d640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:32.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.772+0000 7f247338d640 1 -- 172.21.15.120:0/3254704083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2450009ee0 msgr2=0x7f2450005b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:32.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.772+0000 7f247338d640 1 --2- 172.21.15.120:0/3254704083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2450009ee0 0x7f2450005b70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f246c0aa6a0 tx=0x7f246402c650 comp rx=0 tx=0).stop 2024-02-06T21:40:32.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.772+0000 7f247338d640 1 -- 172.21.15.120:0/3254704083 shutdown_connections 2024-02-06T21:40:32.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.772+0000 7f247338d640 1 --2- 172.21.15.120:0/3254704083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2450009ee0 0x7f2450005b70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:32.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.772+0000 7f247338d640 1 --2- 172.21.15.120:0/3254704083 >> v2:172.21.15.135:3300/0 conn(0x7f24500062a0 0x7f2450009860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:32.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.772+0000 7f247338d640 1 -- 172.21.15.120:0/3254704083 >> 172.21.15.120:0/3254704083 conn(0x7f246c0b8660 msgr2=0x7f246c0b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:32.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.772+0000 7f247338d640 1 -- 172.21.15.120:0/3254704083 shutdown_connections 2024-02-06T21:40:32.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.772+0000 7f247338d640 1 -- 172.21.15.120:0/3254704083 wait complete. 2024-02-06T21:40:32.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.773+0000 7f247338d640 10 monclient: build_initial_monmap 2024-02-06T21:40:32.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.773+0000 7f247338d640 10 monclient: monmap: 2024-02-06T21:40:32.779 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:40:32.779 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:40:32.779 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:40:32.779 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:40:32.779 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:32.779 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:32.779 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:40:32.779 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:40:32.779 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:40:32.779 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:40:32.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.773+0000 7f247338d640 1 librados: starting msgr at 2024-02-06T21:40:32.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.773+0000 7f247338d640 1 librados: starting objecter 2024-02-06T21:40:32.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.773+0000 7f247338d640 1 Processor -- start 2024-02-06T21:40:32.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.774+0000 7f247338d640 1 -- start start 2024-02-06T21:40:32.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.774+0000 7f247338d640 1 librados: setting wanted keys 2024-02-06T21:40:32.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.774+0000 7f247338d640 1 librados: calling monclient init 2024-02-06T21:40:32.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.774+0000 7f247338d640 10 monclient: init 2024-02-06T21:40:32.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.774+0000 7f247338d640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:32.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.774+0000 7f247338d640 10 monclient: _add_conns ranks=[0,2,1] 2024-02-06T21:40:32.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.774+0000 7f247338d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24500062a0 0x7f246c0b7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:32.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.774+0000 7f247338d640 10 monclient(hunting): picked mon.noname-a con 0x7f24500062a0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:40:32.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.774+0000 7f247338d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2450009ee0 0x7f246c13e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:32.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.774+0000 7f247338d640 10 monclient(hunting): picked mon.noname-c con 0x7f2450009ee0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:32.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.774+0000 7f247338d640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2450013f30 0x7f246c1371b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:32.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.775+0000 7f247338d640 10 monclient(hunting): picked mon.noname-b con 0x7f2450013f30 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:40:32.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.775+0000 7f247338d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:32.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.775+0000 7f247338d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246c1cae30 con 0x7f24500062a0 2024-02-06T21:40:32.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.775+0000 7f247338d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:32.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.775+0000 7f247338d640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246c1cab00 con 0x7f2450013f30 2024-02-06T21:40:32.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.775+0000 7f247338d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:32.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.775+0000 7f247338d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246c0b7b10 con 0x7f2450009ee0 2024-02-06T21:40:32.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.775+0000 7f247338d640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:32.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.775+0000 7f247338d640 10 monclient(hunting): authenticate will time out at 1263.112614s 2024-02-06T21:40:32.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.780+0000 7f2471129640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24500062a0 0x7f246c0b7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:32.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.781+0000 7f2471129640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24500062a0 0x7f246c0b7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:43730/0 (socket says 172.21.15.120:43730) 2024-02-06T21:40:32.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.781+0000 7f2471129640 1 -- 172.21.15.120:0/2137751068 learned_addr learned my addr 172.21.15.120:0/2137751068 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:40:32.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.781+0000 7f2471129640 10 monclient(hunting): get_auth_request con 0x7f24500062a0 auth_method 0 2024-02-06T21:40:32.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.781+0000 7f2471129640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:32.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.781+0000 7f2471129640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:32.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.781+0000 7f2471129640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:32.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.783+0000 7f2471129640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:32.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.783+0000 7f2471129640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:32.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.783+0000 7f2471129640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:32.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.787+0000 7f2471129640 10 monclient(hunting): handle_auth_done global_id 4110 payload 824 2024-02-06T21:40:32.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.787+0000 7f2471129640 1 -- 172.21.15.120:0/2137751068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2450009ee0 msgr2=0x7f246c13e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:32.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.787+0000 7f2471129640 1 --2- 172.21.15.120:0/2137751068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2450009ee0 0x7f246c13e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:32.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.787+0000 7f2471129640 1 -- 172.21.15.120:0/2137751068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2450013f30 msgr2=0x7f246c1371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:32.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.787+0000 7f2471129640 1 --2- 172.21.15.120:0/2137751068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2450013f30 0x7f246c1371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:32.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.787+0000 7f2471129640 10 monclient: _finish_hunting 0 2024-02-06T21:40:32.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.787+0000 7f2471129640 1 monclient: found mon.noname-a 2024-02-06T21:40:32.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.787+0000 7f2471129640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:32.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.787+0000 7f2471129640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:40:32.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.787+0000 7f2471129640 1 -- 172.21.15.120:0/2137751068 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2464009950 con 0x7f24500062a0 2024-02-06T21:40:32.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.789+0000 7f2471129640 1 --2- 172.21.15.120:0/2137751068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24500062a0 0x7f246c0b7510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f246800b860 tx=0x7f246800bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:32.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.789+0000 7f2461ffb640 10 client.?.objecter ms_handle_connect 0x7f24500062a0 2024-02-06T21:40:32.791 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.789+0000 7f2461ffb640 10 client.?.objecter resend_mon_ops 2024-02-06T21:40:32.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.792+0000 7f2461ffb640 1 -- 172.21.15.120:0/2137751068 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246800cb70 con 0x7f24500062a0 2024-02-06T21:40:32.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.792+0000 7f2461ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:32.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.792+0000 7f2461ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:40:32.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.792+0000 7f2461ffb640 10 monclient: dump: 2024-02-06T21:40:32.796 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:40:32.796 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:32.796 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:32.796 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:32.796 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:32.796 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:32.796 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:32.796 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.792+0000 7f2461ffb640 10 monclient: _finish_auth 0 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.792+0000 7f2461ffb640 10 monclient: _check_auth_tickets 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.792+0000 7f2461ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.792+0000 7f2461ffb640 1 -- 172.21.15.120:0/2137751068 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246800ccd0 con 0x7f24500062a0 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.792+0000 7f247338d640 5 monclient: authenticate success, global_id 4110 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.792+0000 7f247338d640 10 monclient: _renew_subs 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.793+0000 7f247338d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.793+0000 7f247338d640 1 -- 172.21.15.120:0/2137751068 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246c0b7d30 con 0x7f24500062a0 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.793+0000 7f2461ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.793+0000 7f2461ffb640 1 -- 172.21.15.120:0/2137751068 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2468016670 con 0x7f24500062a0 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.793+0000 7f2461ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.793+0000 7f2461ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.793+0000 7f2461ffb640 10 monclient: dump: 2024-02-06T21:40:32.797 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:40:32.798 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:32.798 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:32.798 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:32.798 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:32.798 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:32.798 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:32.798 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:32.798 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:32.798 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:40:32.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.793+0000 7f24637fe640 4 set_mon_vals no callback set 2024-02-06T21:40:32.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.793+0000 7f247338d640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:40:32.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.793+0000 7f247338d640 10 monclient: _renew_subs 2024-02-06T21:40:32.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.793+0000 7f247338d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:32.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.793+0000 7f247338d640 1 -- 172.21.15.120:0/2137751068 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f246c1378d0 con 0x7f24500062a0 2024-02-06T21:40:32.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.793+0000 7f247338d640 1 librados: init done 2024-02-06T21:40:32.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.794+0000 7f247338d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:32.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.794+0000 7f247338d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:32.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.794+0000 7f247338d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:32.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.794+0000 7f247338d640 1 -- 172.21.15.120:0/2137751068 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f246c1cafc0 con 0x7f24500062a0 2024-02-06T21:40:32.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.977+0000 7f2461ffb640 1 -- 172.21.15.120:0/2137751068 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2468012020 con 0x7f24500062a0 2024-02-06T21:40:32.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.977+0000 7f2461ffb640 1 -- 172.21.15.120:0/2137751068 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f24680060d0 con 0x7f24500062a0 2024-02-06T21:40:32.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.977+0000 7f2461ffb640 10 client.4110.objecter ms_dispatch 0x7f246c000b90 osd_map(1..1 src has 1..1) v4 2024-02-06T21:40:32.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.978+0000 7f2461ffb640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-02-06T21:40:32.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.978+0000 7f2461ffb640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-02-06T21:40:32.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.978+0000 7f2461ffb640 20 client.4110.objecter dump_active .. 0 homeless 2024-02-06T21:40:32.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.978+0000 7f2461ffb640 1 -- 172.21.15.120:0/2137751068 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f246801b060 con 0x7f24500062a0 2024-02-06T21:40:32.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.978+0000 7f2461ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:32.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:32.978+0000 7f2461ffb640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:40:33.047 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.046+0000 7f247338d640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-02-06T21:40:33.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.046+0000 7f247338d640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-02-06T21:40:33.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.046+0000 7f247338d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:33.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.046+0000 7f247338d640 1 -- 172.21.15.120:0/2137751068 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f246c04ef90 con 0x7f24500062a0 2024-02-06T21:40:33.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.206+0000 7f2461ffb640 1 -- 172.21.15.120:0/2137751068 <== mon.0 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2468024020 con 0x7f24500062a0 2024-02-06T21:40:33.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.206+0000 7f2461ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-02-06T21:40:33.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.206+0000 7f2461ffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-02-06T21:40:33.208 INFO:teuthology.orchestra.run.smithi120.stderr:adjusted tunables profile to default 2024-02-06T21:40:33.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.206+0000 7f247338d640 10 librados: watch_flush enter 2024-02-06T21:40:33.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.206+0000 7f247338d640 10 librados: watch_flush exit 2024-02-06T21:40:33.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.206+0000 7f247338d640 20 client.4110.objecter shutdown clearing up homeless session... 2024-02-06T21:40:33.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.206+0000 7f247338d640 10 client.4110.objecter successfully canceled tick 2024-02-06T21:40:33.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.207+0000 7f247338d640 10 monclient: shutdown 2024-02-06T21:40:33.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.207+0000 7f247338d640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:33.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.207+0000 7f247338d640 1 -- 172.21.15.120:0/2137751068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24500062a0 msgr2=0x7f246c0b7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:33.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.207+0000 7f247338d640 1 --2- 172.21.15.120:0/2137751068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24500062a0 0x7f246c0b7510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f246800b860 tx=0x7f246800bd20 comp rx=0 tx=0).stop 2024-02-06T21:40:33.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.207+0000 7f247338d640 1 -- 172.21.15.120:0/2137751068 shutdown_connections 2024-02-06T21:40:33.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.207+0000 7f247338d640 1 --2- 172.21.15.120:0/2137751068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2450013f30 0x7f246c1371b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:33.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.207+0000 7f247338d640 1 --2- 172.21.15.120:0/2137751068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2450009ee0 0x7f246c13e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:33.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.207+0000 7f247338d640 1 --2- 172.21.15.120:0/2137751068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24500062a0 0x7f246c0b7510 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:33.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.207+0000 7f247338d640 1 -- 172.21.15.120:0/2137751068 >> 172.21.15.120:0/2137751068 conn(0x7f246c0b8660 msgr2=0x7f246c06d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:33.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.207+0000 7f247338d640 1 -- 172.21.15.120:0/2137751068 shutdown_connections 2024-02-06T21:40:33.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.207+0000 7f247338d640 1 -- 172.21.15.120:0/2137751068 wait complete. 2024-02-06T21:40:33.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:33.207+0000 7f247338d640 1 librados: shutdown 2024-02-06T21:40:33.219 INFO:tasks.ceph:check_enable_crimson: False 2024-02-06T21:40:33.220 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-02-06T21:40:33.220 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:40:33.220 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-02-06T21:40:33.249 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:40:33.249 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-02-06T21:40:33.317 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:40:33.317 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-02-06T21:40:33.348 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:33.347+0000 7f05e902c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-06T21:40:33.384 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:40:33.384 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-02-06T21:40:33.400 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:33.398+0000 7f4683038040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-06T21:40:33.400 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:33.399+0000 7f05e902c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-06T21:40:33.426 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:40:33.426 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-02-06T21:40:33.453 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:40:33.453 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-02-06T21:40:33.463 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:33.462+0000 7f4683038040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-06T21:40:33.520 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:40:33.520 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-02-06T21:40:33.586 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-02-06T21:40:33.586 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-02-06T21:40:33.653 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 4bd0f591-2cf1-42da-8ac1-dce3188580c0 0 2024-02-06T21:40:33.793 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.792+0000 7fddcdefb640 10 monclient: get_monmap_and_config 2024-02-06T21:40:33.794 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.792+0000 7fddcdefb640 10 monclient: build_initial_monmap 2024-02-06T21:40:33.794 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.792+0000 7fddcdefb640 10 monclient: monmap: 2024-02-06T21:40:33.794 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:33.794 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:33.794 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:40:33.793410+0000 2024-02-06T21:40:33.794 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:40:33.793410+0000 2024-02-06T21:40:33.794 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:33.794 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:33.794 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:40:33.794 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:40:33.795 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:40:33.795 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:40:33.795 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:40:33.795 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:33.797 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.793+0000 7fddcdefb640 1 Processor -- start 2024-02-06T21:40:33.797 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.793+0000 7fddcdefb640 1 -- start start 2024-02-06T21:40:33.797 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.793+0000 7fddcdefb640 10 monclient: init 2024-02-06T21:40:33.797 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.794+0000 7fddcdefb640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:33.797 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.794+0000 7fddcdefb640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-02-06T21:40:33.797 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.794+0000 7fddcdefb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc80b54b0 0x7fddc80a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:33.797 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.794+0000 7fddcdefb640 10 monclient(hunting): picked mon.noname-b con 0x7fddc80b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:33.797 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.794+0000 7fddcdefb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fddc813f830 addr v1:172.21.15.120:6789/0 2024-02-06T21:40:33.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.794+0000 7fddcdefb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fddc8142680 addr v1:172.21.15.135:6789/0 2024-02-06T21:40:33.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.794+0000 7fddcdefb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc8143200 con 0x7fddc813f830 2024-02-06T21:40:33.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.794+0000 7fddcdefb640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc80ab720 con 0x7fddc8142680 2024-02-06T21:40:33.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.794+0000 7fddcdefb640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:33.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.794+0000 7fddcdefb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc80ab520 con 0x7fddc80b54b0 2024-02-06T21:40:33.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.794+0000 7fddcdefb640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:33.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.794+0000 7fddcdefb640 10 monclient(hunting): authenticate will time out at 1275.116215s 2024-02-06T21:40:33.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.794+0000 7fddc7fff640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fddc8142680 0x7fddc8142af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:54368/0 (socket says 172.21.15.135:54368) 2024-02-06T21:40:33.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.794+0000 7fddc7fff640 1 -- 172.21.15.135:0/3014886958 learned_addr learned my addr 172.21.15.135:0/3014886958 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-02-06T21:40:33.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.794+0000 7fddc77fe640 1 --2- 172.21.15.135:0/3014886958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc80b54b0 0x7fddc80a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:33.798 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.795+0000 7fddc77fe640 10 monclient(hunting): get_auth_request con 0x7fddc80b54b0 auth_method 0 2024-02-06T21:40:33.799 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.795+0000 7fddc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:33.799 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.795+0000 7fddc77fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:33.799 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.795+0000 7fddc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:33.799 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.796+0000 7fddc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:33.799 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.796+0000 7fddc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:33.799 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.796+0000 7fddc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:33.800 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc67fc640 1 -- 172.21.15.135:0/3014886958 <== mon.0 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096140965 0 0) 0x7fddc8143200 con 0x7fddc813f830 2024-02-06T21:40:33.800 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc67fc640 10 monclient(hunting): _negotiate 2024-02-06T21:40:33.800 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc67fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:33.801 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:33.801 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc67fc640 10 monclient(hunting): my global_id is 4113 2024-02-06T21:40:33.801 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc67fc640 1 -- 172.21.15.135:0/3014886958 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb0003520 con 0x7fddc813f830 2024-02-06T21:40:33.801 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc67fc640 1 -- 172.21.15.135:0/3014886958 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214749374 0 0) 0x7fddc80ab720 con 0x7fddc8142680 2024-02-06T21:40:33.801 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc67fc640 10 monclient(hunting): _negotiate 2024-02-06T21:40:33.801 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc67fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:33.801 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:33.801 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc67fc640 10 monclient(hunting): my global_id is 4111 2024-02-06T21:40:33.801 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc67fc640 1 -- 172.21.15.135:0/3014886958 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc8143200 con 0x7fddc8142680 2024-02-06T21:40:33.801 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc77fe640 10 monclient(hunting): handle_auth_done global_id 4103 payload 274 2024-02-06T21:40:33.801 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc77fe640 1 -- 172.21.15.135:0/3014886958 >> v1:172.21.15.135:6789/0 conn(0x7fddc8142680 legacy=0x7fddc8142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:33.802 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc77fe640 1 -- 172.21.15.135:0/3014886958 >> v1:172.21.15.120:6789/0 conn(0x7fddc813f830 legacy=0x7fddc813fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:33.802 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc77fe640 10 monclient: _finish_hunting 0 2024-02-06T21:40:33.802 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc77fe640 1 monclient: found mon.noname-b 2024-02-06T21:40:33.802 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:33.802 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:40:33.802 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.797+0000 7fddc77fe640 1 -- 172.21.15.135:0/3014886958 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc80b4800 con 0x7fddc80b54b0 2024-02-06T21:40:33.802 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.799+0000 7fddc77fe640 1 --2- 172.21.15.135:0/3014886958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc80b54b0 0x7fddc80a9860 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fddbc0097d0 tx=0x7fddbc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20f99da443d32287 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:33.806 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddc67fc640 1 -- 172.21.15.135:0/3014886958 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddbc031890 con 0x7fddc80b54b0 2024-02-06T21:40:33.806 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:33.806 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddc67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:40:33.807 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddc67fc640 10 monclient: dump: 2024-02-06T21:40:33.807 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:33.807 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:33.807 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:33.807 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:33.807 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:33.807 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:33.807 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:33.807 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:33.807 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:33.807 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:33.807 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddc67fc640 10 monclient: _finish_auth 0 2024-02-06T21:40:33.808 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddc67fc640 10 monclient: _check_auth_tickets 2024-02-06T21:40:33.808 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:33.808 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddc67fc640 1 -- 172.21.15.135:0/3014886958 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddbc0319f0 con 0x7fddc80b54b0 2024-02-06T21:40:33.808 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddc67fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:33.808 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddc67fc640 1 -- 172.21.15.135:0/3014886958 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddbc031d80 con 0x7fddc80b54b0 2024-02-06T21:40:33.808 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:33.808 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:40:33.808 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddc67fc640 10 monclient: dump: 2024-02-06T21:40:33.808 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:33.808 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:33.808 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:33.809 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:33.809 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:33.809 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:33.809 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:33.809 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:33.809 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:33.809 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:33.809 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddcdefb640 5 monclient: authenticate success, global_id 4103 2024-02-06T21:40:33.809 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddcdefb640 10 monclient: get_monmap_and_config success 2024-02-06T21:40:33.809 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.805+0000 7fddcdefb640 4 set_mon_vals no callback set 2024-02-06T21:40:33.813 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.806+0000 7fddcdefb640 10 monclient: shutdown 2024-02-06T21:40:33.813 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.806+0000 7fddcdefb640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:33.813 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.806+0000 7fddcdefb640 1 -- 172.21.15.135:0/3014886958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc80b54b0 msgr2=0x7fddc80a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:33.813 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.806+0000 7fddcdefb640 1 --2- 172.21.15.135:0/3014886958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc80b54b0 0x7fddc80a9860 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fddbc0097d0 tx=0x7fddbc02b930 comp rx=0 tx=0).stop 2024-02-06T21:40:33.813 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.806+0000 7fddcdefb640 1 -- 172.21.15.135:0/3014886958 shutdown_connections 2024-02-06T21:40:33.814 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.806+0000 7fddcdefb640 1 --2- 172.21.15.135:0/3014886958 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc80b54b0 0x7fddc80a9860 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:33.814 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.806+0000 7fddcdefb640 1 -- 172.21.15.135:0/3014886958 >> 172.21.15.135:0/3014886958 conn(0x7fddc80b8660 msgr2=0x7fddc80b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:33.814 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.806+0000 7fddcdefb640 1 -- 172.21.15.135:0/3014886958 shutdown_connections 2024-02-06T21:40:33.814 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.806+0000 7fddcdefb640 1 -- 172.21.15.135:0/3014886958 wait complete. 2024-02-06T21:40:33.814 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.807+0000 7fddcdefb640 10 monclient: build_initial_monmap 2024-02-06T21:40:33.814 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.807+0000 7fddcdefb640 10 monclient: monmap: 2024-02-06T21:40:33.814 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:33.814 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:40:33.814 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2024-02-06T21:40:33.814 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2024-02-06T21:40:33.814 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:33.814 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:33.814 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:40:33.815 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:40:33.815 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:40:33.815 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:33.815 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.807+0000 7fddcdefb640 1 librados: starting msgr at 2024-02-06T21:40:33.815 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.807+0000 7fddcdefb640 1 librados: starting objecter 2024-02-06T21:40:33.815 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.807+0000 7fddcdefb640 1 Processor -- start 2024-02-06T21:40:33.815 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.807+0000 7fddcdefb640 1 -- start start 2024-02-06T21:40:33.815 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.807+0000 7fddcdefb640 1 librados: setting wanted keys 2024-02-06T21:40:33.815 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.807+0000 7fddcdefb640 1 librados: calling monclient init 2024-02-06T21:40:33.815 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.807+0000 7fddcdefb640 10 monclient: init 2024-02-06T21:40:33.815 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:33.816 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T21:40:33.816 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc80b54b0 0x7fddc80b75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:33.816 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 10 monclient(hunting): picked mon.noname-c con 0x7fddc80b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:33.816 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddc813f830 0x7fddc8134130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:33.816 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 10 monclient(hunting): picked mon.noname-b con 0x7fddc813f830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:40:33.816 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc8142680 0x7fddc8134670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:33.816 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 10 monclient(hunting): picked mon.noname-a con 0x7fddc8142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:40:33.816 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:33.816 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc80ac0a0 con 0x7fddc8142680 2024-02-06T21:40:33.816 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:33.816 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc8134bb0 con 0x7fddc813f830 2024-02-06T21:40:33.817 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:33.817 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc8137d40 con 0x7fddc80b54b0 2024-02-06T21:40:33.817 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:33.817 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddcdefb640 10 monclient(hunting): authenticate will time out at 1275.130333s 2024-02-06T21:40:33.817 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddc6ffd640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddc813f830 0x7fddc8134130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:33.817 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.808+0000 7fddc7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc8142680 0x7fddc8134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:33.817 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc6ffd640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddc813f830 0x7fddc8134130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:47406/0 (socket says 172.21.15.135:47406) 2024-02-06T21:40:33.817 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc6ffd640 1 -- 172.21.15.135:0/2082060707 learned_addr learned my addr 172.21.15.135:0/2082060707 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-02-06T21:40:33.817 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc6ffd640 10 monclient(hunting): get_auth_request con 0x7fddc813f830 auth_method 0 2024-02-06T21:40:33.817 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:33.817 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc6ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:33.817 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:33.817 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc7fff640 10 monclient(hunting): get_auth_request con 0x7fddc8142680 auth_method 0 2024-02-06T21:40:33.817 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:33.818 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc7fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:33.818 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:33.818 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:33.818 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:33.818 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:33.818 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:33.818 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:33.818 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.809+0000 7fddc7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:33.818 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.812+0000 7fddc6ffd640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-02-06T21:40:33.818 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.812+0000 7fddc6ffd640 1 -- 172.21.15.135:0/2082060707 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc80b54b0 msgr2=0x7fddc80b75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:33.818 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.812+0000 7fddc6ffd640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc80b54b0 0x7fddc80b75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:33.819 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.812+0000 7fddc6ffd640 1 -- 172.21.15.135:0/2082060707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc8142680 msgr2=0x7fddc8134670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:33.819 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.812+0000 7fddc6ffd640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc8142680 0x7fddc8134670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:33.819 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.812+0000 7fddc6ffd640 10 monclient: _finish_hunting 0 2024-02-06T21:40:33.819 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.812+0000 7fddc6ffd640 1 monclient: found mon.noname-b 2024-02-06T21:40:33.819 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.812+0000 7fddc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:33.819 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.812+0000 7fddc6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:40:33.819 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.812+0000 7fddc6ffd640 1 -- 172.21.15.135:0/2082060707 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc009400 con 0x7fddc813f830 2024-02-06T21:40:33.821 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.814+0000 7fddc6ffd640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddc813f830 0x7fddc8134130 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fddb800c460 tx=0x7fddb800c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:33.821 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.814+0000 7fddaffff640 10 client.?.objecter ms_handle_connect 0x7fddc813f830 2024-02-06T21:40:33.821 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.814+0000 7fddaffff640 10 client.?.objecter resend_mon_ops 2024-02-06T21:40:33.821 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddaffff640 1 -- 172.21.15.135:0/2082060707 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb80052d0 con 0x7fddc813f830 2024-02-06T21:40:33.821 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:33.821 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddaffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:40:33.821 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddaffff640 10 monclient: dump: 2024-02-06T21:40:33.822 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:33.822 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:33.822 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:33.822 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:33.822 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:33.822 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:33.822 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:33.822 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:33.822 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:33.822 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:33.822 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddaffff640 10 monclient: _finish_auth 0 2024-02-06T21:40:33.822 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddaffff640 10 monclient: _check_auth_tickets 2024-02-06T21:40:33.823 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:33.823 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddaffff640 1 -- 172.21.15.135:0/2082060707 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb8005430 con 0x7fddc813f830 2024-02-06T21:40:33.823 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddaffff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:33.823 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddaffff640 1 -- 172.21.15.135:0/2082060707 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8012850 con 0x7fddc813f830 2024-02-06T21:40:33.823 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:33.823 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddaffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:40:33.823 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddaffff640 10 monclient: dump: 2024-02-06T21:40:33.823 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:33.823 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:33.823 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:33.824 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:33.824 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:33.824 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:33.824 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:33.824 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:33.824 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:33.824 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:33.824 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddcdefb640 5 monclient: authenticate success, global_id 4114 2024-02-06T21:40:33.824 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddcdefb640 10 monclient: _renew_subs 2024-02-06T21:40:33.825 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddcdefb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:33.825 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddcdefb640 1 -- 172.21.15.135:0/2082060707 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc8138260 con 0x7fddc813f830 2024-02-06T21:40:33.825 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddc5ffb640 4 set_mon_vals no callback set 2024-02-06T21:40:33.825 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddcdefb640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:40:33.825 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddcdefb640 10 monclient: _renew_subs 2024-02-06T21:40:33.825 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddcdefb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:33.825 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddcdefb640 1 -- 172.21.15.135:0/2082060707 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc81374b0 con 0x7fddc813f830 2024-02-06T21:40:33.825 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.818+0000 7fddcdefb640 1 librados: init done 2024-02-06T21:40:33.825 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.819+0000 7fddcdefb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:33.825 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.819+0000 7fddcdefb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:33.825 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.819+0000 7fddcdefb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:33.826 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.819+0000 7fddcdefb640 1 -- 172.21.15.135:0/2082060707 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc804ef90 con 0x7fddc813f830 2024-02-06T21:40:33.937 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:33.936+0000 7f05e902c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-06T21:40:33.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddc6ffd640 1 -- 172.21.15.135:0/2082060707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddc813f830 msgr2=0x7fddc8134130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-02-06T21:40:33.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddc6ffd640 1 -- 172.21.15.135:0/2082060707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddc813f830 msgr2=0x7fddc8134130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-06T21:40:33.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddc6ffd640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddc813f830 0x7fddc8134130 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fddb800c460 tx=0x7fddb800c920 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-06T21:40:33.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddc6ffd640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddc813f830 0x7fddc8134130 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fddb800c460 tx=0x7fddb800c920 comp rx=0 tx=0).stop 2024-02-06T21:40:33.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:40:33.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:33.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 1 -- 172.21.15.135:0/2082060707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddc813f830 msgr2=0x7fddc8134130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-06T21:40:33.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddc813f830 0x7fddc8134130 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:33.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T21:40:33.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddb4005240 0x7fddb4007630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:33.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 10 monclient(hunting): picked mon.b con 0x7fddb4005240 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:40:33.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc80b54b0 0x7fddb400a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:33.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 10 monclient(hunting): picked mon.c con 0x7fddc80b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:33.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc8142680 0x7fddb400cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:33.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 10 monclient(hunting): picked mon.a con 0x7fddc8142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:40:33.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:33.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 1 -- 172.21.15.135:0/2082060707 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb400d5c0 con 0x7fddc8142680 2024-02-06T21:40:33.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:33.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 1 -- 172.21.15.135:0/2082060707 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb400d700 con 0x7fddb4005240 2024-02-06T21:40:33.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:33.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 1 -- 172.21.15.135:0/2082060707 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb400d870 con 0x7fddc80b54b0 2024-02-06T21:40:33.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:33.991+0000 7fddaffff640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:33.995 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:33.992+0000 7f05e902c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-06T21:40:34.001 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.001+0000 7fddc77fe640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddb4005240 0x7fddb4007630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:34.002 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.001+0000 7fddc77fe640 10 monclient(hunting): get_auth_request con 0x7fddb4005240 auth_method 0 2024-02-06T21:40:34.002 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.001+0000 7fddc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:34.002 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.001+0000 7fddc77fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:34.002 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.001+0000 7fddc77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-02-06T21:40:34.002 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.001+0000 7fddc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:34.002 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.001+0000 7fddc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:34.002 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.001+0000 7fddc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-02-06T21:40:34.003 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.002+0000 7fddc6ffd640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc80b54b0 0x7fddb400a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:34.003 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc6ffd640 10 monclient(hunting): get_auth_request con 0x7fddc80b54b0 auth_method 0 2024-02-06T21:40:34.003 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:34.004 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc6ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:34.004 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-02-06T21:40:34.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 10 monclient(hunting): handle_auth_done global_id 4114 payload 293 2024-02-06T21:40:34.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 1 -- 172.21.15.135:0/2082060707 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc80b54b0 msgr2=0x7fddb400a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:34.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc80b54b0 0x7fddb400a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 1 -- 172.21.15.135:0/2082060707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc8142680 msgr2=0x7fddb400cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:34.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc8142680 0x7fddb400cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 10 monclient: _finish_hunting 0 2024-02-06T21:40:34.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 1 monclient: found mon.b 2024-02-06T21:40:34.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:34.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:34.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 1 -- 172.21.15.135:0/2082060707 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fddb400db50 con 0x7fddb4005240 2024-02-06T21:40:34.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:34.005 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:34.006 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 1 -- 172.21.15.135:0/2082060707 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc80b4800 con 0x7fddb4005240 2024-02-06T21:40:34.006 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 10 monclient: _finish_auth 0 2024-02-06T21:40:34.006 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 10 monclient: _check_auth_tickets 2024-02-06T21:40:34.006 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.003+0000 7fddc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:34.006 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.005+0000 7fddc77fe640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddb4005240 0x7fddb4007630 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fddbc031790 tx=0x7fddb8007d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:34.006 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.005+0000 7fddaffff640 10 client.4114.objecter ms_handle_connect 0x7fddb4005240 2024-02-06T21:40:34.006 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.005+0000 7fddaffff640 10 client.4114.objecter resend_mon_ops 2024-02-06T21:40:34.016 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:34.015+0000 7f4683038040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-06T21:40:34.039 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:34.038+0000 7f05e902c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-06T21:40:34.072 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:34.071+0000 7f4683038040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-06T21:40:34.094 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:34.094+0000 7f05e902c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-06T21:40:34.110 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddaffff640 1 -- 172.21.15.135:0/2082060707 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddbc046080 con 0x7fddb4005240 2024-02-06T21:40:34.110 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:34.110 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddaffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:40:34.110 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddaffff640 10 monclient: dump: 2024-02-06T21:40:34.110 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:34.110 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddaffff640 1 -- 172.21.15.135:0/2082060707 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddbc031b60 con 0x7fddb4005240 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddaffff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddaffff640 1 -- 172.21.15.135:0/2082060707 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fddbc03a7d0 con 0x7fddb4005240 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddaffff640 1 -- 172.21.15.135:0/2082060707 <== mon.1 v2:172.21.15.135:3300/0 4 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fddbc043850 con 0x7fddb4005240 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddaffff640 10 client.4114.objecter ms_dispatch 0x7fddc8000b90 osd_map(2..2 src has 1..2) v4 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddaffff640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddaffff640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2024-02-06T21:40:34.111 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddc57fa640 4 set_mon_vals no callback set 2024-02-06T21:40:34.112 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddaffff640 20 client.4114.objecter dump_active .. 0 homeless 2024-02-06T21:40:34.112 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddaffff640 1 -- 172.21.15.135:0/2082060707 <== mon.1 v2:172.21.15.135:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fddbc035020 con 0x7fddb4005240 2024-02-06T21:40:34.112 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddaffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:34.112 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.109+0000 7fddaffff640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:40:34.117 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:34.116+0000 7f4683038040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-06T21:40:34.139 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:34.138+0000 7f05e902c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-06T21:40:34.174 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:34.173+0000 7f4683038040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-06T21:40:34.199 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.198+0000 7fddcdefb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4bd0f591-2cf1-42da-8ac1-dce3188580c0", "id": 0}] 2024-02-06T21:40:34.199 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.198+0000 7fddcdefb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4bd0f591-2cf1-42da-8ac1-dce3188580c0", "id": 0}] 2024-02-06T21:40:34.199 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.198+0000 7fddcdefb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:34.199 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.198+0000 7fddcdefb640 1 -- 172.21.15.135:0/2082060707 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4bd0f591-2cf1-42da-8ac1-dce3188580c0", "id": 0} v 0) v1 -- 0x7fddc8137f00 con 0x7fddb4005240 2024-02-06T21:40:34.216 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:34.215+0000 7f4683038040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-06T21:40:34.258 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:34.257+0000 7f05e902c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-06T21:40:34.327 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:34.326+0000 7f05e902c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-06T21:40:34.337 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:34.335+0000 7f4683038040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-06T21:40:34.373 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:34.373+0000 7f05e902c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-06T21:40:34.411 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:34.410+0000 7f4683038040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-06T21:40:34.428 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:34.427+0000 7f05e902c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-06T21:40:34.455 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.454+0000 7fddaffff640 1 -- 172.21.15.135:0/2082060707 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4bd0f591-2cf1-42da-8ac1-dce3188580c0", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fddbc043400 con 0x7fddb4005240 2024-02-06T21:40:34.455 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.454+0000 7fddaffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4bd0f591-2cf1-42da-8ac1-dce3188580c0", "id": 0}] 2024-02-06T21:40:34.455 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.454+0000 7fddaffff640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:40:34.456 INFO:teuthology.orchestra.run.smithi135.stdout:0 2024-02-06T21:40:34.457 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 10 librados: watch_flush enter 2024-02-06T21:40:34.457 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 10 librados: watch_flush exit 2024-02-06T21:40:34.457 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 20 client.4114.objecter shutdown clearing up homeless session... 2024-02-06T21:40:34.457 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 10 client.4114.objecter successfully canceled tick 2024-02-06T21:40:34.457 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 10 monclient: shutdown 2024-02-06T21:40:34.457 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:34.458 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 1 -- 172.21.15.135:0/2082060707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddb4005240 msgr2=0x7fddb4007630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:34.458 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddb4005240 0x7fddb4007630 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fddbc031790 tx=0x7fddb8007d70 comp rx=0 tx=0).stop 2024-02-06T21:40:34.458 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 1 -- 172.21.15.135:0/2082060707 shutdown_connections 2024-02-06T21:40:34.458 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddb4005240 0x7fddb4007630 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.458 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddc8142680 0x7fddb400cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.458 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 1 --2- 172.21.15.135:0/2082060707 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddc80b54b0 0x7fddb400a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.458 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 1 -- 172.21.15.135:0/2082060707 >> 172.21.15.135:0/2082060707 conn(0x7fddc80b8660 msgr2=0x7fddc80744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:34.458 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 1 -- 172.21.15.135:0/2082060707 shutdown_connections 2024-02-06T21:40:34.458 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 1 -- 172.21.15.135:0/2082060707 wait complete. 2024-02-06T21:40:34.458 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.455+0000 7fddcdefb640 1 librados: shutdown 2024-02-06T21:40:34.459 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:34.455+0000 7f4683038040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-06T21:40:34.467 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 81dc801c-3768-4c2c-9adb-6c8300a341f5 1 2024-02-06T21:40:34.479 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:34.478+0000 7f05e902c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-06T21:40:34.508 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:34.507+0000 7f4683038040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-06T21:40:34.555 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:34.553+0000 7f4683038040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-06T21:40:34.566 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.565+0000 7f7a5acba640 10 monclient: get_monmap_and_config 2024-02-06T21:40:34.566 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.565+0000 7f7a5acba640 10 monclient: build_initial_monmap 2024-02-06T21:40:34.566 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.565+0000 7f7a5acba640 10 monclient: monmap: 2024-02-06T21:40:34.566 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:34.566 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:34.566 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:40:34.565759+0000 2024-02-06T21:40:34.566 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:40:34.565759+0000 2024-02-06T21:40:34.566 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:34.566 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:34.567 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:40:34.567 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:40:34.567 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:40:34.567 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:40:34.567 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:40:34.567 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:34.570 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.565+0000 7f7a5acba640 1 Processor -- start 2024-02-06T21:40:34.570 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.565+0000 7f7a5acba640 1 -- start start 2024-02-06T21:40:34.570 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.565+0000 7f7a5acba640 10 monclient: init 2024-02-06T21:40:34.570 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.566+0000 7f7a5acba640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:34.570 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.566+0000 7f7a5acba640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-02-06T21:40:34.570 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.566+0000 7f7a5acba640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7a540b54b0 0x7f7a540a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:34.570 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.566+0000 7f7a5acba640 10 monclient(hunting): picked mon.noname-c con 0x7f7a540b54b0 addr v2:172.21.15.135:3300/0 2024-02-06T21:40:34.570 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.566+0000 7f7a5acba640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a5413f830 addr v1:172.21.15.135:6789/0 2024-02-06T21:40:34.570 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.566+0000 7f7a5acba640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a54142680 0x7f7a54142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:34.570 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.566+0000 7f7a5acba640 10 monclient(hunting): picked mon.noname-b con 0x7f7a54142680 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:34.571 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.566+0000 7f7a5acba640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a540ab720 con 0x7f7a5413f830 2024-02-06T21:40:34.571 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.566+0000 7f7a5acba640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:34.571 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.566+0000 7f7a5acba640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a540ab520 con 0x7f7a540b54b0 2024-02-06T21:40:34.571 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.566+0000 7f7a5acba640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:34.571 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.566+0000 7f7a5acba640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a540b5890 con 0x7f7a54142680 2024-02-06T21:40:34.571 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.566+0000 7f7a5acba640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:34.571 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.566+0000 7f7a5acba640 10 monclient(hunting): authenticate will time out at 1275.888440s 2024-02-06T21:40:34.571 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.567+0000 7f7a58a56640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7a540b54b0 0x7f7a540a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:34.571 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.567+0000 7f7a59257640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a54142680 0x7f7a54142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:34.571 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.567+0000 7f7a58a56640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7a540b54b0 0x7f7a540a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:47416/0 (socket says 172.21.15.135:47416) 2024-02-06T21:40:34.571 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.567+0000 7f7a58a56640 1 -- 172.21.15.135:0/2833036304 learned_addr learned my addr 172.21.15.135:0/2833036304 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-02-06T21:40:34.571 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.567+0000 7f7a58a56640 10 monclient(hunting): get_auth_request con 0x7f7a540b54b0 auth_method 0 2024-02-06T21:40:34.572 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.567+0000 7f7a58a56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:34.572 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.567+0000 7f7a58a56640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:34.572 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.567+0000 7f7a58a56640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:34.572 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.567+0000 7f7a59257640 10 monclient(hunting): get_auth_request con 0x7f7a54142680 auth_method 0 2024-02-06T21:40:34.572 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.567+0000 7f7a59257640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:34.572 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.567+0000 7f7a59257640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:34.572 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.567+0000 7f7a59257640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:34.572 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.568+0000 7f7a58a56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:34.572 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.568+0000 7f7a58a56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:34.572 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.568+0000 7f7a58a56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:34.572 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.568+0000 7f7a59257640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:34.572 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.568+0000 7f7a59257640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:34.573 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.568+0000 7f7a59257640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:34.573 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.569+0000 7f7a53fff640 1 -- 172.21.15.135:0/2833036304 >> v1:172.21.15.135:6789/0 conn(0x7f7a5413f830 legacy=0x7f7a5413fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-02-06T21:40:34.573 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.569+0000 7f7a53fff640 1 -- 172.21.15.135:0/2833036304 >> v1:172.21.15.135:6789/0 conn(0x7f7a5413f830 legacy=0x7f7a5413fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-06T21:40:34.573 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.569+0000 7f7a53fff640 1 --1- 172.21.15.135:0/2833036304 >> v1:172.21.15.135:6789/0 conn(0x7f7a5413f830 0x7f7a5413fc70 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-02-06T21:40:34.574 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.570+0000 7f7a59257640 10 monclient(hunting): handle_auth_done global_id 4106 payload 274 2024-02-06T21:40:34.574 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.570+0000 7f7a59257640 1 -- 172.21.15.135:0/2833036304 >> v2:172.21.15.135:3300/0 conn(0x7f7a540b54b0 msgr2=0x7f7a540a9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:34.574 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.570+0000 7f7a59257640 1 --2- 172.21.15.135:0/2833036304 >> v2:172.21.15.135:3300/0 conn(0x7f7a540b54b0 0x7f7a540a9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.574 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.570+0000 7f7a59257640 1 -- 172.21.15.135:0/2833036304 >> v1:172.21.15.135:6789/0 conn(0x7f7a5413f830 legacy=0x7f7a5413fc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-06T21:40:34.574 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.570+0000 7f7a59257640 10 monclient: _finish_hunting 0 2024-02-06T21:40:34.574 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.570+0000 7f7a59257640 1 monclient: found mon.noname-b 2024-02-06T21:40:34.574 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.570+0000 7f7a59257640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:34.575 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.570+0000 7f7a59257640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:40:34.575 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.570+0000 7f7a59257640 1 -- 172.21.15.135:0/2833036304 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a540b4800 con 0x7f7a54142680 2024-02-06T21:40:34.575 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.572+0000 7f7a59257640 1 --2- 172.21.15.135:0/2833036304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a54142680 0x7f7a54142ac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009d90 tx=0x7f7a4c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb5848d534cdd01c server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:34.576 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a537fe640 1 -- 172.21.15.135:0/2833036304 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c02cbc0 con 0x7f7a54142680 2024-02-06T21:40:34.577 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:34.577 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:40:34.577 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a537fe640 10 monclient: dump: 2024-02-06T21:40:34.577 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:34.577 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:34.577 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:34.577 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:34.577 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:34.577 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:34.577 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:34.578 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:34.578 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:34.578 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:34.578 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a537fe640 10 monclient: _finish_auth 0 2024-02-06T21:40:34.578 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a537fe640 10 monclient: _check_auth_tickets 2024-02-06T21:40:34.578 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:34.578 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a537fe640 1 -- 172.21.15.135:0/2833036304 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a4c02cd20 con 0x7f7a54142680 2024-02-06T21:40:34.578 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a537fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:34.578 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a5acba640 5 monclient: authenticate success, global_id 4106 2024-02-06T21:40:34.578 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a537fe640 1 -- 172.21.15.135:0/2833036304 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c034660 con 0x7f7a54142680 2024-02-06T21:40:34.578 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a5acba640 10 monclient: get_monmap_and_config success 2024-02-06T21:40:34.578 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a5acba640 4 set_mon_vals no callback set 2024-02-06T21:40:34.578 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:34.578 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:40:34.578 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.575+0000 7f7a537fe640 10 monclient: dump: 2024-02-06T21:40:34.579 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:34.579 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:34.579 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:34.579 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:34.579 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:34.579 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:34.579 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:34.579 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:34.579 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:34.579 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:34.582 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.576+0000 7f7a5acba640 10 monclient: shutdown 2024-02-06T21:40:34.583 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.576+0000 7f7a5acba640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:34.583 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.576+0000 7f7a5acba640 1 -- 172.21.15.135:0/2833036304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a54142680 msgr2=0x7f7a54142ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:34.583 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.576+0000 7f7a5acba640 1 --2- 172.21.15.135:0/2833036304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a54142680 0x7f7a54142ac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009d90 tx=0x7f7a4c02be10 comp rx=0 tx=0).stop 2024-02-06T21:40:34.583 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.576+0000 7f7a5acba640 1 -- 172.21.15.135:0/2833036304 shutdown_connections 2024-02-06T21:40:34.583 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.576+0000 7f7a5acba640 1 --2- 172.21.15.135:0/2833036304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a54142680 0x7f7a54142ac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.583 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.576+0000 7f7a5acba640 1 --2- 172.21.15.135:0/2833036304 >> v2:172.21.15.135:3300/0 conn(0x7f7a540b54b0 0x7f7a540a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.583 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.576+0000 7f7a5acba640 1 -- 172.21.15.135:0/2833036304 >> 172.21.15.135:0/2833036304 conn(0x7f7a540b8660 msgr2=0x7f7a540b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:34.583 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.576+0000 7f7a5acba640 1 -- 172.21.15.135:0/2833036304 shutdown_connections 2024-02-06T21:40:34.583 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.576+0000 7f7a5acba640 1 -- 172.21.15.135:0/2833036304 wait complete. 2024-02-06T21:40:34.583 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.577+0000 7f7a5acba640 10 monclient: build_initial_monmap 2024-02-06T21:40:34.583 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.577+0000 7f7a5acba640 10 monclient: monmap: 2024-02-06T21:40:34.583 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:34.583 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:40:34.583 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2024-02-06T21:40:34.584 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2024-02-06T21:40:34.584 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:34.584 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:34.584 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:40:34.584 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:40:34.584 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:40:34.584 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:34.584 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.577+0000 7f7a5acba640 1 librados: starting msgr at 2024-02-06T21:40:34.584 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.577+0000 7f7a5acba640 1 librados: starting objecter 2024-02-06T21:40:34.584 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.577+0000 7f7a5acba640 1 Processor -- start 2024-02-06T21:40:34.584 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.577+0000 7f7a5acba640 1 -- start start 2024-02-06T21:40:34.584 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.577+0000 7f7a5acba640 1 librados: setting wanted keys 2024-02-06T21:40:34.584 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.577+0000 7f7a5acba640 1 librados: calling monclient init 2024-02-06T21:40:34.585 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.577+0000 7f7a5acba640 10 monclient: init 2024-02-06T21:40:34.585 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:34.585 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 10 monclient: _add_conns ranks=[0,2,1] 2024-02-06T21:40:34.585 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a540b54b0 0x7f7a54136510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:34.585 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 10 monclient(hunting): picked mon.noname-a con 0x7f7a540b54b0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:40:34.585 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a5413f830 0x7f7a541385b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:34.585 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 10 monclient(hunting): picked mon.noname-c con 0x7f7a5413f830 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:34.585 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a54142680 0x7f7a54138af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:34.585 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 10 monclient(hunting): picked mon.noname-b con 0x7f7a54142680 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:40:34.585 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:34.585 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a54137120 con 0x7f7a540b54b0 2024-02-06T21:40:34.585 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:34.585 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a54137cb0 con 0x7f7a54142680 2024-02-06T21:40:34.585 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:34.585 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a541390b0 con 0x7f7a5413f830 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.578+0000 7f7a5acba640 10 monclient(hunting): authenticate will time out at 1275.900657s 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a59257640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a54142680 0x7f7a54138af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a58a56640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a540b54b0 0x7f7a54136510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a53fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a5413f830 0x7f7a541385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a59257640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a54142680 0x7f7a54138af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:47420/0 (socket says 172.21.15.135:47420) 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a59257640 1 -- 172.21.15.135:0/665768911 learned_addr learned my addr 172.21.15.135:0/665768911 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a59257640 10 monclient(hunting): get_auth_request con 0x7f7a54142680 auth_method 0 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a59257640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a59257640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a59257640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a58a56640 10 monclient(hunting): get_auth_request con 0x7f7a540b54b0 auth_method 0 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a58a56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a58a56640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a58a56640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:34.586 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a53fff640 10 monclient(hunting): get_auth_request con 0x7f7a5413f830 auth_method 0 2024-02-06T21:40:34.587 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:34.587 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a53fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:34.587 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.579+0000 7f7a53fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:34.587 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.580+0000 7f7a59257640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:34.587 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.580+0000 7f7a59257640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:34.587 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.580+0000 7f7a59257640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:34.587 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.580+0000 7f7a53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:34.587 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.580+0000 7f7a53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:34.587 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.580+0000 7f7a53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:34.587 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.580+0000 7f7a58a56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:34.587 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.580+0000 7f7a58a56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:34.587 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.580+0000 7f7a58a56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:34.588 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.582+0000 7f7a59257640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-02-06T21:40:34.588 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.582+0000 7f7a59257640 1 -- 172.21.15.135:0/665768911 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a5413f830 msgr2=0x7f7a541385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:34.588 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.582+0000 7f7a59257640 1 --2- 172.21.15.135:0/665768911 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a5413f830 0x7f7a541385b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.588 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.582+0000 7f7a59257640 1 -- 172.21.15.135:0/665768911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a540b54b0 msgr2=0x7f7a54136510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:34.589 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.582+0000 7f7a59257640 1 --2- 172.21.15.135:0/665768911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a540b54b0 0x7f7a54136510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.589 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.582+0000 7f7a59257640 10 monclient: _finish_hunting 0 2024-02-06T21:40:34.589 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.582+0000 7f7a59257640 1 monclient: found mon.noname-b 2024-02-06T21:40:34.589 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.582+0000 7f7a59257640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:34.589 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.582+0000 7f7a59257640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:40:34.589 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.582+0000 7f7a59257640 1 -- 172.21.15.135:0/665768911 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c009a70 con 0x7f7a54142680 2024-02-06T21:40:34.589 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.584+0000 7f7a59257640 1 --2- 172.21.15.135:0/665768911 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a54142680 0x7f7a54138af0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009e80 tx=0x7f7a4c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:34.589 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.584+0000 7f7a517fa640 10 client.?.objecter ms_handle_connect 0x7f7a54142680 2024-02-06T21:40:34.589 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.584+0000 7f7a517fa640 10 client.?.objecter resend_mon_ops 2024-02-06T21:40:34.591 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a517fa640 1 -- 172.21.15.135:0/665768911 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c03e610 con 0x7f7a54142680 2024-02-06T21:40:34.591 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:34.591 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a517fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:40:34.592 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a517fa640 10 monclient: dump: 2024-02-06T21:40:34.592 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:34.592 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:34.592 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:34.592 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:34.592 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:34.592 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:34.592 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:34.592 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:34.592 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:34.592 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:34.592 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a517fa640 10 monclient: _finish_auth 0 2024-02-06T21:40:34.592 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a517fa640 10 monclient: _check_auth_tickets 2024-02-06T21:40:34.592 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:34.593 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a517fa640 1 -- 172.21.15.135:0/665768911 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a4c02ce90 con 0x7f7a54142680 2024-02-06T21:40:34.593 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a517fa640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:34.593 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a517fa640 1 -- 172.21.15.135:0/665768911 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c03e9b0 con 0x7f7a54142680 2024-02-06T21:40:34.593 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:34.593 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:40:34.593 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a517fa640 10 monclient: dump: 2024-02-06T21:40:34.593 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:34.593 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:34.593 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:34.593 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:34.593 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:34.593 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:34.593 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:34.593 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:34.593 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:34.594 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:34.594 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a5acba640 5 monclient: authenticate success, global_id 4120 2024-02-06T21:40:34.594 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a5acba640 10 monclient: _renew_subs 2024-02-06T21:40:34.594 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a5acba640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:34.594 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a52ffd640 4 set_mon_vals no callback set 2024-02-06T21:40:34.594 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a5acba640 1 -- 172.21.15.135:0/665768911 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a54138190 con 0x7f7a54142680 2024-02-06T21:40:34.594 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a5acba640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:40:34.594 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a5acba640 10 monclient: _renew_subs 2024-02-06T21:40:34.594 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a5acba640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:34.595 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a5acba640 1 -- 172.21.15.135:0/665768911 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a54135c50 con 0x7f7a54142680 2024-02-06T21:40:34.595 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.590+0000 7f7a5acba640 1 librados: init done 2024-02-06T21:40:34.596 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.591+0000 7f7a5acba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:34.596 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.591+0000 7f7a5acba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:34.596 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.591+0000 7f7a5acba640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:34.596 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.591+0000 7f7a5acba640 1 -- 172.21.15.135:0/665768911 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a54135930 con 0x7f7a54142680 2024-02-06T21:40:34.596 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.592+0000 7f7a517fa640 1 -- 172.21.15.135:0/665768911 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7a4c03b050 con 0x7f7a54142680 2024-02-06T21:40:34.695 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.694+0000 7f7a517fa640 1 -- 172.21.15.135:0/665768911 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f7a4c049450 con 0x7f7a54142680 2024-02-06T21:40:34.695 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.694+0000 7f7a517fa640 10 client.4120.objecter ms_dispatch 0x7f7a54000b90 osd_map(3..3 src has 1..3) v4 2024-02-06T21:40:34.696 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.694+0000 7f7a517fa640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2024-02-06T21:40:34.696 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.694+0000 7f7a517fa640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2024-02-06T21:40:34.696 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.694+0000 7f7a517fa640 20 client.4120.objecter dump_active .. 0 homeless 2024-02-06T21:40:34.697 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.694+0000 7f7a517fa640 1 -- 172.21.15.135:0/665768911 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7a4c04e060 con 0x7f7a54142680 2024-02-06T21:40:34.697 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.694+0000 7f7a517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:34.697 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.694+0000 7f7a517fa640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:40:34.769 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.767+0000 7f7a5acba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "81dc801c-3768-4c2c-9adb-6c8300a341f5", "id": 1}] 2024-02-06T21:40:34.769 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.767+0000 7f7a5acba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "81dc801c-3768-4c2c-9adb-6c8300a341f5", "id": 1}] 2024-02-06T21:40:34.769 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.768+0000 7f7a5acba640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:34.769 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.768+0000 7f7a5acba640 1 -- 172.21.15.135:0/665768911 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "81dc801c-3768-4c2c-9adb-6c8300a341f5", "id": 1} v 0) v1 -- 0x7f7a5404ef90 con 0x7f7a54142680 2024-02-06T21:40:34.848 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.847+0000 7f7a517fa640 1 -- 172.21.15.135:0/665768911 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "81dc801c-3768-4c2c-9adb-6c8300a341f5", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7a4c057020 con 0x7f7a54142680 2024-02-06T21:40:34.848 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.847+0000 7f7a517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "81dc801c-3768-4c2c-9adb-6c8300a341f5", "id": 1}] 2024-02-06T21:40:34.848 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.847+0000 7f7a517fa640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:40:34.848 INFO:teuthology.orchestra.run.smithi135.stdout:1 2024-02-06T21:40:34.850 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.847+0000 7f7a5acba640 10 librados: watch_flush enter 2024-02-06T21:40:34.850 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.848+0000 7f7a5acba640 10 librados: watch_flush exit 2024-02-06T21:40:34.850 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.848+0000 7f7a5acba640 20 client.4120.objecter shutdown clearing up homeless session... 2024-02-06T21:40:34.850 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.848+0000 7f7a5acba640 10 client.4120.objecter successfully canceled tick 2024-02-06T21:40:34.850 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.848+0000 7f7a5acba640 10 monclient: shutdown 2024-02-06T21:40:34.850 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.848+0000 7f7a5acba640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:34.850 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.848+0000 7f7a5acba640 1 -- 172.21.15.135:0/665768911 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a54142680 msgr2=0x7f7a54138af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:34.850 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.848+0000 7f7a5acba640 1 --2- 172.21.15.135:0/665768911 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a54142680 0x7f7a54138af0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009e80 tx=0x7f7a4c004a40 comp rx=0 tx=0).stop 2024-02-06T21:40:34.850 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.848+0000 7f7a5acba640 1 -- 172.21.15.135:0/665768911 shutdown_connections 2024-02-06T21:40:34.850 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.848+0000 7f7a5acba640 1 --2- 172.21.15.135:0/665768911 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a54142680 0x7f7a54138af0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.851 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.848+0000 7f7a5acba640 1 --2- 172.21.15.135:0/665768911 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a5413f830 0x7f7a541385b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.851 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.848+0000 7f7a5acba640 1 --2- 172.21.15.135:0/665768911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a540b54b0 0x7f7a54136510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.851 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.848+0000 7f7a5acba640 1 -- 172.21.15.135:0/665768911 >> 172.21.15.135:0/665768911 conn(0x7f7a540b8660 msgr2=0x7f7a5413a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:34.851 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.848+0000 7f7a5acba640 1 -- 172.21.15.135:0/665768911 shutdown_connections 2024-02-06T21:40:34.851 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.848+0000 7f7a5acba640 1 -- 172.21.15.135:0/665768911 wait complete. 2024-02-06T21:40:34.851 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.848+0000 7f7a5acba640 1 librados: shutdown 2024-02-06T21:40:34.860 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 9289721b-c13f-4a17-be68-3698090f69af 2 2024-02-06T21:40:34.878 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:34.877+0000 7f05e902c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-06T21:40:34.951 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:34.949+0000 7f4683038040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-06T21:40:34.979 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.976+0000 7f494fc81640 10 monclient: get_monmap_and_config 2024-02-06T21:40:34.980 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.976+0000 7f494fc81640 10 monclient: build_initial_monmap 2024-02-06T21:40:34.980 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.976+0000 7f494fc81640 10 monclient: monmap: 2024-02-06T21:40:34.980 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:34.980 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:34.980 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:40:34.976705+0000 2024-02-06T21:40:34.980 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:40:34.976705+0000 2024-02-06T21:40:34.980 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:34.980 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:34.980 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:40:34.980 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:40:34.980 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:40:34.981 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:40:34.981 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:40:34.981 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:34.981 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.976+0000 7f494fc81640 1 Processor -- start 2024-02-06T21:40:34.981 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.976+0000 7f494fc81640 1 -- start start 2024-02-06T21:40:34.981 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.976+0000 7f494fc81640 10 monclient: init 2024-02-06T21:40:34.981 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.977+0000 7f494fc81640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:34.981 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.977+0000 7f494fc81640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-02-06T21:40:34.981 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.977+0000 7f494fc81640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49480b54b0 addr v1:172.21.15.120:6789/0 2024-02-06T21:40:34.981 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.977+0000 7f494fc81640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f494813ff40 0x7f49480a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:34.981 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.977+0000 7f494fc81640 10 monclient(hunting): picked mon.noname-b con 0x7f494813ff40 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:34.981 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.977+0000 7f494fc81640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4948142680 0x7f4948142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:34.981 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.977+0000 7f494fc81640 10 monclient(hunting): picked mon.noname-a con 0x7f4948142680 addr v2:172.21.15.120:3300/0 2024-02-06T21:40:34.981 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.977+0000 7f494fc81640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49480ab720 con 0x7f49480b54b0 2024-02-06T21:40:34.981 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.977+0000 7f494fc81640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:34.982 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.977+0000 7f494fc81640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49480ab520 con 0x7f4948142680 2024-02-06T21:40:34.982 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.977+0000 7f494fc81640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:34.982 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.977+0000 7f494fc81640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49480b5890 con 0x7f494813ff40 2024-02-06T21:40:34.982 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.977+0000 7f494fc81640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:34.982 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.977+0000 7f494fc81640 10 monclient(hunting): authenticate will time out at 1276.299492s 2024-02-06T21:40:34.982 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.978+0000 7f494d21c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f494813ff40 0x7f49480a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:34.982 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.978+0000 7f494e21e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4948142680 0x7f4948142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:34.982 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.978+0000 7f494d21c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f494813ff40 0x7f49480a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.135:37112/0 (socket says 172.21.15.135:37112) 2024-02-06T21:40:34.982 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.978+0000 7f494d21c640 1 -- 172.21.15.135:0/929357696 learned_addr learned my addr 172.21.15.135:0/929357696 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-02-06T21:40:34.982 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.978+0000 7f494d21c640 10 monclient(hunting): get_auth_request con 0x7f494813ff40 auth_method 0 2024-02-06T21:40:34.982 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.978+0000 7f494d21c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:34.982 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.978+0000 7f494d21c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:34.982 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.978+0000 7f494d21c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:34.982 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.978+0000 7f494e21e640 10 monclient(hunting): get_auth_request con 0x7f4948142680 auth_method 0 2024-02-06T21:40:34.983 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.978+0000 7f494e21e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:34.983 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.978+0000 7f494e21e640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:34.983 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.978+0000 7f494e21e640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:34.983 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.979+0000 7f494d21c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:34.983 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.979+0000 7f494d21c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:34.983 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.979+0000 7f494d21c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:34.983 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.979+0000 7f494e21e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:34.983 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.979+0000 7f494e21e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:34.983 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.979+0000 7f494e21e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:34.984 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.980+0000 7f494d21c640 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2024-02-06T21:40:34.984 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.980+0000 7f494d21c640 1 -- 172.21.15.135:0/929357696 >> v2:172.21.15.120:3300/0 conn(0x7f4948142680 msgr2=0x7f4948142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:34.984 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.980+0000 7f494d21c640 1 --2- 172.21.15.135:0/929357696 >> v2:172.21.15.120:3300/0 conn(0x7f4948142680 0x7f4948142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.984 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.980+0000 7f494d21c640 1 -- 172.21.15.135:0/929357696 >> v1:172.21.15.120:6789/0 conn(0x7f49480b54b0 legacy=0x7f494813f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:34.984 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.980+0000 7f494d21c640 10 monclient: _finish_hunting 0 2024-02-06T21:40:34.984 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.980+0000 7f494d21c640 1 monclient: found mon.noname-b 2024-02-06T21:40:34.984 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.980+0000 7f494d21c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:34.984 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.980+0000 7f494d21c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:40:34.984 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.980+0000 7f494d21c640 1 -- 172.21.15.135:0/929357696 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49480b4800 con 0x7f494813ff40 2024-02-06T21:40:34.984 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.982+0000 7f494d21c640 1 --2- 172.21.15.135:0/929357696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f494813ff40 0x7f49480a9860 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4944009870 tx=0x7f494402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34321919b954485b server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:34.986 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494d21c640 1 -- 172.21.15.135:0/929357696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f494813ff40 msgr2=0x7f49480a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-02-06T21:40:34.986 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494d21c640 1 -- 172.21.15.135:0/929357696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f494813ff40 msgr2=0x7f49480a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-06T21:40:34.986 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494d21c640 1 --2- 172.21.15.135:0/929357696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f494813ff40 0x7f49480a9860 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4944009870 tx=0x7f494402b8f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-06T21:40:34.986 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494d21c640 1 --2- 172.21.15.135:0/929357696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f494813ff40 0x7f49480a9860 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4944009870 tx=0x7f494402b8f0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.986 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494ca1b640 10 monclient: ms_handle_reset current mon [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:34.986 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494ca1b640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:34.986 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494ca1b640 1 -- 172.21.15.135:0/929357696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f494813ff40 msgr2=0x7f49480a9860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-06T21:40:34.986 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494ca1b640 1 --2- 172.21.15.135:0/929357696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f494813ff40 0x7f49480a9860 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:34.986 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494ca1b640 10 monclient: _add_conns ranks=[4,1] 2024-02-06T21:40:34.987 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494ca1b640 1 --2- 172.21.15.135:0/929357696 >> v2:172.21.15.135:3300/0 conn(0x7f4934003500 0x7f4934005980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:34.987 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494ca1b640 10 monclient(hunting): picked mon.noname-c con 0x7f4934003500 addr v2:172.21.15.135:3300/0 2024-02-06T21:40:34.987 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494ca1b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f49340062b0 addr v1:172.21.15.135:6789/0 2024-02-06T21:40:34.987 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494ca1b640 1 -- 172.21.15.135:0/929357696 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493400a220 con 0x7f49340062b0 2024-02-06T21:40:34.987 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494ca1b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:34.987 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494ca1b640 1 -- 172.21.15.135:0/929357696 --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f493400b420 con 0x7f4934003500 2024-02-06T21:40:34.987 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494ca1b640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:34.987 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494ca1b640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:34.987 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.984+0000 7f494da1d640 1 --2- 172.21.15.135:0/929357696 >> v2:172.21.15.135:3300/0 conn(0x7f4934003500 0x7f4934005980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:34.987 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.985+0000 7f494da1d640 10 monclient(hunting): get_auth_request con 0x7f4934003500 auth_method 0 2024-02-06T21:40:34.987 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.985+0000 7f494da1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:34.987 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.985+0000 7f494da1d640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:34.987 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.985+0000 7f494da1d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-02-06T21:40:34.988 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.986+0000 7f494da1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:34.988 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.986+0000 7f494da1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:34.988 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.986+0000 7f494da1d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-02-06T21:40:34.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.990+0000 7f494da1d640 10 monclient(hunting): handle_auth_done global_id 4112 payload 293 2024-02-06T21:40:34.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.990+0000 7f494da1d640 1 -- 172.21.15.135:0/929357696 >> v1:172.21.15.135:6789/0 conn(0x7f49340062b0 legacy=0x7f4934008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:34.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.990+0000 7f494da1d640 10 monclient: _finish_hunting 0 2024-02-06T21:40:34.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.990+0000 7f494da1d640 1 monclient: found mon.noname-c 2024-02-06T21:40:34.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.990+0000 7f494da1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:34.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.990+0000 7f494da1d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T21:40:34.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.990+0000 7f494da1d640 1 -- 172.21.15.135:0/929357696 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493400b650 con 0x7f4934003500 2024-02-06T21:40:34.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.994+0000 7f494da1d640 1 --2- 172.21.15.135:0/929357696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4934003500 0x7f4934005980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f49380015a0 tx=0x7f4944009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98c3b09fcbf5567e server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:34.999 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494ca1b640 1 -- 172.21.15.135:0/929357696 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49380050d0 con 0x7f4934003500 2024-02-06T21:40:34.999 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494ca1b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:35.000 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494ca1b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:40:35.000 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494ca1b640 10 monclient: dump: 2024-02-06T21:40:35.000 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:35.000 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:35.000 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.000 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.000 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:35.000 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.000 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:35.000 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:35.000 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:35.001 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.001 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494ca1b640 10 monclient: _finish_auth 0 2024-02-06T21:40:35.001 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494ca1b640 10 monclient: _check_auth_tickets 2024-02-06T21:40:35.001 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494ca1b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:35.001 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494ca1b640 1 -- 172.21.15.135:0/929357696 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4938005230 con 0x7f4934003500 2024-02-06T21:40:35.001 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494fc81640 5 monclient: authenticate success, global_id 4112 2024-02-06T21:40:35.001 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494fc81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:40:35.001 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494ca1b640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:35.001 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494ca1b640 1 -- 172.21.15.135:0/929357696 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493800c7b0 con 0x7f4934003500 2024-02-06T21:40:35.001 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494fc81640 10 monclient: get_monmap_and_config success 2024-02-06T21:40:35.001 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494fc81640 4 set_mon_vals no callback set 2024-02-06T21:40:35.001 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494ca1b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:35.002 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494ca1b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:40:35.002 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.998+0000 7f494ca1b640 10 monclient: dump: 2024-02-06T21:40:35.002 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:35.002 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:35.002 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.002 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.002 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:35.002 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.002 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:35.002 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:35.002 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:35.003 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.010 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.999+0000 7f494fc81640 10 monclient: shutdown 2024-02-06T21:40:35.010 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.999+0000 7f494fc81640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:35.010 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.999+0000 7f494fc81640 1 -- 172.21.15.135:0/929357696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4934003500 msgr2=0x7f4934005980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:35.010 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.999+0000 7f494fc81640 1 --2- 172.21.15.135:0/929357696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4934003500 0x7f4934005980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f49380015a0 tx=0x7f4944009240 comp rx=0 tx=0).stop 2024-02-06T21:40:35.010 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.999+0000 7f494e21e640 1 -- 172.21.15.135:0/929357696 reap_dead start 2024-02-06T21:40:35.011 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.999+0000 7f494fc81640 1 -- 172.21.15.135:0/929357696 shutdown_connections 2024-02-06T21:40:35.011 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.999+0000 7f494fc81640 1 --2- 172.21.15.135:0/929357696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4934003500 0x7f4934005980 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.011 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.999+0000 7f494fc81640 1 -- 172.21.15.135:0/929357696 >> 172.21.15.135:0/929357696 conn(0x7f49480b8660 msgr2=0x7f49480b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:35.011 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.999+0000 7f494fc81640 1 -- 172.21.15.135:0/929357696 shutdown_connections 2024-02-06T21:40:35.011 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:34.999+0000 7f494fc81640 1 -- 172.21.15.135:0/929357696 wait complete. 2024-02-06T21:40:35.011 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.000+0000 7f494fc81640 10 monclient: build_initial_monmap 2024-02-06T21:40:35.011 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.000+0000 7f494fc81640 10 monclient: monmap: 2024-02-06T21:40:35.011 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:35.011 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:40:35.011 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2024-02-06T21:40:35.012 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2024-02-06T21:40:35.012 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:35.012 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.012 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:40:35.012 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:40:35.012 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:40:35.012 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.012 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.000+0000 7f494fc81640 1 librados: starting msgr at 2024-02-06T21:40:35.012 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.000+0000 7f494fc81640 1 librados: starting objecter 2024-02-06T21:40:35.012 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.000+0000 7f494fc81640 1 Processor -- start 2024-02-06T21:40:35.012 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.000+0000 7f494fc81640 1 -- start start 2024-02-06T21:40:35.012 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.000+0000 7f494fc81640 1 librados: setting wanted keys 2024-02-06T21:40:35.012 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.000+0000 7f494fc81640 1 librados: calling monclient init 2024-02-06T21:40:35.012 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.000+0000 7f494fc81640 10 monclient: init 2024-02-06T21:40:35.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:35.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T21:40:35.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4934003500 0x7f4948139060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:35.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 10 monclient(hunting): picked mon.noname-b con 0x7f4934003500 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:40:35.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49481365d0 0x7f4948136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:35.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 10 monclient(hunting): picked mon.noname-c con 0x7f49481365d0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:35.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4948132280 0x7f4948132700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:35.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 10 monclient(hunting): picked mon.noname-a con 0x7f4948132280 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:40:35.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:35.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4948138410 con 0x7f4948132280 2024-02-06T21:40:35.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:35.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4948137920 con 0x7f4934003500 2024-02-06T21:40:35.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:35.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49481395a0 con 0x7f49481365d0 2024-02-06T21:40:35.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:35.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494fc81640 10 monclient(hunting): authenticate will time out at 1276.323360s 2024-02-06T21:40:35.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.001+0000 7f494da1d640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4934003500 0x7f4948139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:35.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494e21e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4948132280 0x7f4948132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:35.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494d21c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49481365d0 0x7f4948136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:35.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494da1d640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4934003500 0x7f4948139060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:47438/0 (socket says 172.21.15.135:47438) 2024-02-06T21:40:35.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494da1d640 1 -- 172.21.15.135:0/115315105 learned_addr learned my addr 172.21.15.135:0/115315105 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-02-06T21:40:35.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494da1d640 10 monclient(hunting): get_auth_request con 0x7f4934003500 auth_method 0 2024-02-06T21:40:35.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494da1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:35.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494da1d640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:35.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494da1d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:35.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494d21c640 10 monclient(hunting): get_auth_request con 0x7f49481365d0 auth_method 0 2024-02-06T21:40:35.014 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494d21c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494d21c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494d21c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494e21e640 10 monclient(hunting): get_auth_request con 0x7f4948132280 auth_method 0 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494e21e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494e21e640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494e21e640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494d21c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.002+0000 7f494d21c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.003+0000 7f494d21c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.003+0000 7f494e21e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.003+0000 7f494e21e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.003+0000 7f494e21e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.003+0000 7f494da1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.003+0000 7f494da1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.003+0000 7f494da1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:35.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.005+0000 7f494d21c640 10 monclient(hunting): handle_auth_done global_id 4115 payload 824 2024-02-06T21:40:35.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.005+0000 7f494d21c640 1 -- 172.21.15.135:0/115315105 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4934003500 msgr2=0x7f4948139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:35.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.005+0000 7f494d21c640 1 --2- 172.21.15.135:0/115315105 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4934003500 0x7f4948139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.005+0000 7f494d21c640 1 -- 172.21.15.135:0/115315105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4948132280 msgr2=0x7f4948132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:35.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.005+0000 7f494d21c640 1 --2- 172.21.15.135:0/115315105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4948132280 0x7f4948132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.005+0000 7f494d21c640 10 monclient: _finish_hunting 0 2024-02-06T21:40:35.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.005+0000 7f494d21c640 1 monclient: found mon.noname-c 2024-02-06T21:40:35.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.005+0000 7f494d21c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:35.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.005+0000 7f494d21c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:40:35.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.005+0000 7f494d21c640 1 -- 172.21.15.135:0/115315105 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4948132c40 con 0x7f49481365d0 2024-02-06T21:40:35.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.007+0000 7f494d21c640 1 --2- 172.21.15.135:0/115315105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49481365d0 0x7f4948136dd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f494806c7b0 tx=0x7f49440046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:35.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.007+0000 7f493e7fc640 10 client.?.objecter ms_handle_connect 0x7f49481365d0 2024-02-06T21:40:35.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.007+0000 7f493e7fc640 10 client.?.objecter resend_mon_ops 2024-02-06T21:40:35.018 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f493e7fc640 1 -- 172.21.15.135:0/115315105 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4944045640 con 0x7f49481365d0 2024-02-06T21:40:35.018 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f493e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:35.018 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f493e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:40:35.018 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f493e7fc640 10 monclient: dump: 2024-02-06T21:40:35.018 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:35.018 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:35.018 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.018 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.018 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:35.018 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.018 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:35.018 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:35.018 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:35.019 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.019 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f493e7fc640 10 monclient: _finish_auth 0 2024-02-06T21:40:35.019 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f493e7fc640 10 monclient: _check_auth_tickets 2024-02-06T21:40:35.019 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f493e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:35.019 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f493e7fc640 1 -- 172.21.15.135:0/115315105 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49440457a0 con 0x7f49481365d0 2024-02-06T21:40:35.019 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f493e7fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:35.019 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f493e7fc640 1 -- 172.21.15.135:0/115315105 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4944045b30 con 0x7f49481365d0 2024-02-06T21:40:35.019 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f493e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:35.019 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f493e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:40:35.019 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f493e7fc640 10 monclient: dump: 2024-02-06T21:40:35.019 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:35.019 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:35.019 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.019 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.020 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:35.020 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.020 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:35.020 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:35.020 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:35.020 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.020 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f494fc81640 5 monclient: authenticate success, global_id 4115 2024-02-06T21:40:35.020 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f494fc81640 10 monclient: _renew_subs 2024-02-06T21:40:35.020 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f493ffff640 4 set_mon_vals no callback set 2024-02-06T21:40:35.020 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f494fc81640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:35.020 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f494fc81640 1 -- 172.21.15.135:0/115315105 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4948137310 con 0x7f49481365d0 2024-02-06T21:40:35.020 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f494fc81640 10 client.4115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:40:35.020 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f494fc81640 10 monclient: _renew_subs 2024-02-06T21:40:35.021 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f494fc81640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:35.021 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f494fc81640 1 -- 172.21.15.135:0/115315105 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49481388d0 con 0x7f49481365d0 2024-02-06T21:40:35.021 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.011+0000 7f494fc81640 1 librados: init done 2024-02-06T21:40:35.021 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.012+0000 7f494fc81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:35.021 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.012+0000 7f494fc81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:35.021 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.012+0000 7f494fc81640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:35.021 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.012+0000 7f494fc81640 1 -- 172.21.15.135:0/115315105 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f494804ef90 con 0x7f49481365d0 2024-02-06T21:40:35.025 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:35.024+0000 7f05e902c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-06T21:40:35.090 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:35.089+0000 7f4683038040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-06T21:40:35.151 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:35.150+0000 7f05e902c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-06T21:40:35.181 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.180+0000 7f493e7fc640 1 -- 172.21.15.135:0/115315105 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f494402f050 con 0x7f49481365d0 2024-02-06T21:40:35.188 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.187+0000 7f493e7fc640 1 -- 172.21.15.135:0/115315105 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f494403bcb0 con 0x7f49481365d0 2024-02-06T21:40:35.188 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.187+0000 7f493e7fc640 10 client.4115.objecter ms_dispatch 0x7f4948000b90 osd_map(4..4 src has 1..4) v4 2024-02-06T21:40:35.188 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.187+0000 7f493e7fc640 3 client.4115.objecter handle_osd_map got epochs [4,4] > 0 2024-02-06T21:40:35.188 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.187+0000 7f493e7fc640 3 client.4115.objecter handle_osd_map decoding full epoch 4 2024-02-06T21:40:35.189 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.187+0000 7f493e7fc640 20 client.4115.objecter dump_active .. 0 homeless 2024-02-06T21:40:35.189 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.187+0000 7f493e7fc640 1 -- 172.21.15.135:0/115315105 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4944035020 con 0x7f49481365d0 2024-02-06T21:40:35.189 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.187+0000 7f493e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:35.189 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.187+0000 7f493e7fc640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:40:35.197 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:35.196+0000 7f05e902c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-06T21:40:35.221 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:35.219+0000 7f4683038040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-06T21:40:35.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.260+0000 7f494fc81640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9289721b-c13f-4a17-be68-3698090f69af", "id": 2}] 2024-02-06T21:40:35.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.260+0000 7f494fc81640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9289721b-c13f-4a17-be68-3698090f69af", "id": 2}] 2024-02-06T21:40:35.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.260+0000 7f494fc81640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:35.261 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.260+0000 7f494fc81640 1 -- 172.21.15.135:0/115315105 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "9289721b-c13f-4a17-be68-3698090f69af", "id": 2} v 0) v1 -- 0x7f494804ef90 con 0x7f49481365d0 2024-02-06T21:40:35.269 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:35.268+0000 7f4683038040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-06T21:40:35.311 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:35.310+0000 7f05e902c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-06T21:40:35.393 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:35.391+0000 7f4683038040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-06T21:40:35.400 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.399+0000 7f493e7fc640 1 -- 172.21.15.135:0/115315105 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9289721b-c13f-4a17-be68-3698090f69af", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f494402d040 con 0x7f49481365d0 2024-02-06T21:40:35.400 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.399+0000 7f493e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9289721b-c13f-4a17-be68-3698090f69af", "id": 2}] 2024-02-06T21:40:35.400 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.399+0000 7f493e7fc640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:40:35.401 INFO:teuthology.orchestra.run.smithi135.stdout:2 2024-02-06T21:40:35.402 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.400+0000 7f494fc81640 10 librados: watch_flush enter 2024-02-06T21:40:35.403 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.400+0000 7f494fc81640 10 librados: watch_flush exit 2024-02-06T21:40:35.403 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.400+0000 7f494fc81640 20 client.4115.objecter shutdown clearing up homeless session... 2024-02-06T21:40:35.403 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.400+0000 7f494fc81640 10 client.4115.objecter successfully canceled tick 2024-02-06T21:40:35.403 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.400+0000 7f494fc81640 10 monclient: shutdown 2024-02-06T21:40:35.403 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.400+0000 7f494fc81640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:35.403 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.400+0000 7f494fc81640 1 -- 172.21.15.135:0/115315105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49481365d0 msgr2=0x7f4948136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:35.403 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.400+0000 7f494fc81640 1 --2- 172.21.15.135:0/115315105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49481365d0 0x7f4948136dd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f494806c7b0 tx=0x7f49440046c0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.403 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.400+0000 7f494fc81640 1 -- 172.21.15.135:0/115315105 shutdown_connections 2024-02-06T21:40:35.403 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.400+0000 7f494fc81640 1 --2- 172.21.15.135:0/115315105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4948132280 0x7f4948132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.403 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.400+0000 7f494fc81640 1 --2- 172.21.15.135:0/115315105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49481365d0 0x7f4948136dd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.403 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.400+0000 7f494fc81640 1 --2- 172.21.15.135:0/115315105 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4934003500 0x7f4948139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.403 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.400+0000 7f494fc81640 1 -- 172.21.15.135:0/115315105 >> 172.21.15.135:0/115315105 conn(0x7f49480b8660 msgr2=0x7f494813f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:35.403 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.401+0000 7f494fc81640 1 -- 172.21.15.135:0/115315105 shutdown_connections 2024-02-06T21:40:35.404 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.401+0000 7f494fc81640 1 -- 172.21.15.135:0/115315105 wait complete. 2024-02-06T21:40:35.404 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.401+0000 7f494fc81640 1 librados: shutdown 2024-02-06T21:40:35.408 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:35.407+0000 7f05e902c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-06T21:40:35.412 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 53534db0-2d4d-44ed-90e8-d67ea878b51a 3 2024-02-06T21:40:35.489 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:35.487+0000 7f4683038040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-06T21:40:35.511 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.510+0000 7f8fb73a8640 10 monclient: get_monmap_and_config 2024-02-06T21:40:35.511 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.510+0000 7f8fb73a8640 10 monclient: build_initial_monmap 2024-02-06T21:40:35.511 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.510+0000 7f8fb73a8640 10 monclient: monmap: 2024-02-06T21:40:35.512 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:35.512 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:35.512 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:40:35.511252+0000 2024-02-06T21:40:35.512 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:40:35.511252+0000 2024-02-06T21:40:35.512 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:35.512 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.512 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:40:35.512 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:40:35.512 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:40:35.512 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:40:35.512 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:40:35.512 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.512 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 1 Processor -- start 2024-02-06T21:40:35.515 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 1 -- start start 2024-02-06T21:40:35.515 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 10 monclient: init 2024-02-06T21:40:35.515 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:35.515 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-02-06T21:40:35.515 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8fb00b54b0 addr v1:172.21.15.135:6789/0 2024-02-06T21:40:35.515 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8fb013ff40 0x7f8fb00a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:35.515 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 10 monclient(hunting): picked mon.noname-a con 0x7f8fb013ff40 addr v2:172.21.15.120:3300/0 2024-02-06T21:40:35.516 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8fb0142680 0x7f8fb0142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:35.516 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 10 monclient(hunting): picked mon.noname-c con 0x7f8fb0142680 addr v2:172.21.15.135:3300/0 2024-02-06T21:40:35.516 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb00ab720 con 0x7f8fb00b54b0 2024-02-06T21:40:35.516 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:35.516 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fb00ab520 con 0x7f8fb013ff40 2024-02-06T21:40:35.516 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:35.516 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fb00b5890 con 0x7f8fb0142680 2024-02-06T21:40:35.517 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:35.517 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.511+0000 7f8fb73a8640 10 monclient(hunting): authenticate will time out at 1276.833947s 2024-02-06T21:40:35.517 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.512+0000 7f8fb5945640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8fb0142680 0x7f8fb0142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:35.517 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.512+0000 7f8fb4943640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8fb013ff40 0x7f8fb00a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:35.517 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.513+0000 7f8fb4943640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8fb013ff40 0x7f8fb00a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.135:37116/0 (socket says 172.21.15.135:37116) 2024-02-06T21:40:35.517 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.513+0000 7f8fb4943640 1 -- 172.21.15.135:0/4043140993 learned_addr learned my addr 172.21.15.135:0/4043140993 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-02-06T21:40:35.517 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.513+0000 7f8fb4943640 10 monclient(hunting): get_auth_request con 0x7f8fb013ff40 auth_method 0 2024-02-06T21:40:35.517 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.513+0000 7f8fb4943640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:35.517 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.513+0000 7f8fb4943640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:35.517 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.513+0000 7f8fb4943640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:35.517 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.513+0000 7f8fb5945640 10 monclient(hunting): get_auth_request con 0x7f8fb0142680 auth_method 0 2024-02-06T21:40:35.517 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.513+0000 7f8fb5945640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:35.517 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.513+0000 7f8fb5945640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:35.518 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.513+0000 7f8fb5945640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:35.518 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.513+0000 7f8fb4943640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:35.518 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.513+0000 7f8fb4943640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:35.518 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.513+0000 7f8fb4943640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:35.518 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.514+0000 7f8fb5945640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:35.518 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.514+0000 7f8fb5945640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:35.518 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.514+0000 7f8fb5945640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:35.519 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.515+0000 7f8fb4943640 10 monclient(hunting): handle_auth_done global_id 4128 payload 274 2024-02-06T21:40:35.519 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.515+0000 7f8fb4943640 1 -- 172.21.15.135:0/4043140993 >> v2:172.21.15.135:3300/0 conn(0x7f8fb0142680 msgr2=0x7f8fb0142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:35.519 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.515+0000 7f8fb4943640 1 --2- 172.21.15.135:0/4043140993 >> v2:172.21.15.135:3300/0 conn(0x7f8fb0142680 0x7f8fb0142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.519 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.515+0000 7f8fb4943640 1 -- 172.21.15.135:0/4043140993 >> v1:172.21.15.135:6789/0 conn(0x7f8fb00b54b0 legacy=0x7f8fb013f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:35.519 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.515+0000 7f8fb4943640 10 monclient: _finish_hunting 0 2024-02-06T21:40:35.519 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.515+0000 7f8fb4943640 1 monclient: found mon.noname-a 2024-02-06T21:40:35.519 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.515+0000 7f8fb4943640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:35.519 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.515+0000 7f8fb4943640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:40:35.519 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.515+0000 7f8fb4943640 1 -- 172.21.15.135:0/4043140993 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb00b4800 con 0x7f8fb013ff40 2024-02-06T21:40:35.519 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.517+0000 7f8fb4943640 1 --2- 172.21.15.135:0/4043140993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fb013ff40 0x7f8fb00a9860 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8fb00aa6a0 tx=0x7f8fa002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c3d0bbca6c9b66 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:35.522 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fa7fff640 1 -- 172.21.15.135:0/4043140993 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0030860 con 0x7f8fb013ff40 2024-02-06T21:40:35.522 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fa7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:35.522 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fa7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:40:35.522 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fa7fff640 10 monclient: dump: 2024-02-06T21:40:35.522 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:35.523 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:35.523 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.523 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.523 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:35.523 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.523 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:35.523 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:35.523 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:35.523 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.523 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fa7fff640 10 monclient: _finish_auth 0 2024-02-06T21:40:35.523 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fa7fff640 10 monclient: _check_auth_tickets 2024-02-06T21:40:35.523 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:35.524 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fa7fff640 1 -- 172.21.15.135:0/4043140993 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa00309c0 con 0x7f8fb013ff40 2024-02-06T21:40:35.524 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fa7fff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:35.524 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fa7fff640 1 -- 172.21.15.135:0/4043140993 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0030d50 con 0x7f8fb013ff40 2024-02-06T21:40:35.524 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fa7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:35.524 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fa7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:40:35.524 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fa7fff640 10 monclient: dump: 2024-02-06T21:40:35.524 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:35.524 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:35.524 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.524 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.524 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:35.525 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.525 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:35.525 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:35.525 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:35.525 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.525 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fb73a8640 5 monclient: authenticate success, global_id 4128 2024-02-06T21:40:35.525 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fb73a8640 10 monclient: get_monmap_and_config success 2024-02-06T21:40:35.525 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.521+0000 7f8fb73a8640 4 set_mon_vals no callback set 2024-02-06T21:40:35.529 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.522+0000 7f8fb73a8640 10 monclient: shutdown 2024-02-06T21:40:35.530 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.522+0000 7f8fb73a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:35.530 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.522+0000 7f8fb73a8640 1 -- 172.21.15.135:0/4043140993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fb013ff40 msgr2=0x7f8fb00a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:35.530 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.522+0000 7f8fb73a8640 1 --2- 172.21.15.135:0/4043140993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fb013ff40 0x7f8fb00a9860 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8fb00aa6a0 tx=0x7f8fa002b9a0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.530 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.522+0000 7f8fb73a8640 1 -- 172.21.15.135:0/4043140993 shutdown_connections 2024-02-06T21:40:35.530 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.522+0000 7f8fb73a8640 1 --2- 172.21.15.135:0/4043140993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fb013ff40 0x7f8fb00a9860 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.530 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.522+0000 7f8fb73a8640 1 --2- 172.21.15.135:0/4043140993 >> v2:172.21.15.135:3300/0 conn(0x7f8fb0142680 0x7f8fb0142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.530 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.522+0000 7f8fb73a8640 1 -- 172.21.15.135:0/4043140993 >> 172.21.15.135:0/4043140993 conn(0x7f8fb00b8660 msgr2=0x7f8fb00b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:35.530 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.522+0000 7f8fb73a8640 1 -- 172.21.15.135:0/4043140993 shutdown_connections 2024-02-06T21:40:35.530 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.522+0000 7f8fb73a8640 1 -- 172.21.15.135:0/4043140993 wait complete. 2024-02-06T21:40:35.530 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.522+0000 7f8fb73a8640 10 monclient: build_initial_monmap 2024-02-06T21:40:35.530 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.522+0000 7f8fb73a8640 10 monclient: monmap: 2024-02-06T21:40:35.530 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:35.530 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:40:35.530 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2024-02-06T21:40:35.531 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2024-02-06T21:40:35.531 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:35.531 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.531 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:40:35.531 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:40:35.531 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:40:35.531 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.531 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 1 librados: starting msgr at 2024-02-06T21:40:35.531 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 1 librados: starting objecter 2024-02-06T21:40:35.531 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 1 Processor -- start 2024-02-06T21:40:35.531 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 1 -- start start 2024-02-06T21:40:35.531 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 1 librados: setting wanted keys 2024-02-06T21:40:35.531 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 1 librados: calling monclient init 2024-02-06T21:40:35.532 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 10 monclient: init 2024-02-06T21:40:35.532 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:35.532 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T21:40:35.532 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fb00b54b0 0x7f8fb0138dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:35.532 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 10 monclient(hunting): picked mon.noname-c con 0x7f8fb00b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:35.532 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8fb013ff40 0x7f8fb01351c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:35.532 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 10 monclient(hunting): picked mon.noname-b con 0x7f8fb013ff40 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:40:35.532 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fb0142680 0x7f8fb0135700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:35.532 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 10 monclient(hunting): picked mon.noname-a con 0x7f8fb0142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:40:35.532 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:35.532 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.523+0000 7f8fb73a8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb0134770 con 0x7f8fb0142680 2024-02-06T21:40:35.532 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb73a8640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:35.532 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb73a8640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb0134a50 con 0x7f8fb013ff40 2024-02-06T21:40:35.532 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb73a8640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:35.533 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb73a8640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb0134d30 con 0x7f8fb00b54b0 2024-02-06T21:40:35.533 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb73a8640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:35.533 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb73a8640 10 monclient(hunting): authenticate will time out at 1276.845965s 2024-02-06T21:40:35.533 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb4943640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8fb013ff40 0x7f8fb01351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:35.533 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb5945640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fb0142680 0x7f8fb0135700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:35.533 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb5144640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fb00b54b0 0x7f8fb0138dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:35.533 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb5945640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fb0142680 0x7f8fb0135700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.135:37120/0 (socket says 172.21.15.135:37120) 2024-02-06T21:40:35.533 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb5945640 1 -- 172.21.15.135:0/1397730557 learned_addr learned my addr 172.21.15.135:0/1397730557 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-02-06T21:40:35.533 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb5945640 10 monclient(hunting): get_auth_request con 0x7f8fb0142680 auth_method 0 2024-02-06T21:40:35.533 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb5945640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:35.533 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb5945640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:35.533 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb5945640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:35.533 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb4943640 10 monclient(hunting): get_auth_request con 0x7f8fb013ff40 auth_method 0 2024-02-06T21:40:35.533 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb4943640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:35.534 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb4943640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:35.534 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.524+0000 7f8fb4943640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:35.534 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.525+0000 7f8fb5144640 10 monclient(hunting): get_auth_request con 0x7f8fb00b54b0 auth_method 0 2024-02-06T21:40:35.534 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.525+0000 7f8fb5144640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:35.534 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.525+0000 7f8fb5144640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:35.534 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.525+0000 7f8fb5144640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:35.534 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.525+0000 7f8fb5945640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:35.534 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.525+0000 7f8fb5945640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:35.534 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.525+0000 7f8fb5945640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:35.534 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.525+0000 7f8fb5144640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:35.534 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.525+0000 7f8fb5144640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:35.534 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.525+0000 7f8fb5144640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:35.534 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.525+0000 7f8fb4943640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:35.535 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.525+0000 7f8fb4943640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:35.535 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.525+0000 7f8fb4943640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:35.535 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.528+0000 7f8fb5144640 10 monclient(hunting): handle_auth_done global_id 4118 payload 824 2024-02-06T21:40:35.535 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.528+0000 7f8fb5144640 1 -- 172.21.15.135:0/1397730557 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8fb013ff40 msgr2=0x7f8fb01351c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:35.535 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.528+0000 7f8fb5144640 1 --2- 172.21.15.135:0/1397730557 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8fb013ff40 0x7f8fb01351c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.535 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.528+0000 7f8fb5144640 1 -- 172.21.15.135:0/1397730557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fb0142680 msgr2=0x7f8fb0135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:35.535 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.528+0000 7f8fb5144640 1 --2- 172.21.15.135:0/1397730557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fb0142680 0x7f8fb0135700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.535 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.528+0000 7f8fb5144640 10 monclient: _finish_hunting 0 2024-02-06T21:40:35.535 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.528+0000 7f8fb5144640 1 monclient: found mon.noname-c 2024-02-06T21:40:35.535 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.528+0000 7f8fb5144640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:35.535 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.528+0000 7f8fb5144640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:40:35.535 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.528+0000 7f8fb5144640 1 -- 172.21.15.135:0/1397730557 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa00094d0 con 0x7f8fb00b54b0 2024-02-06T21:40:35.537 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.530+0000 7f8fb5144640 1 --2- 172.21.15.135:0/1397730557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fb00b54b0 0x7f8fb0138dd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8fac00c340 tx=0x7f8fac00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:35.537 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.530+0000 7f8fa5ffb640 10 client.?.objecter ms_handle_connect 0x7f8fb00b54b0 2024-02-06T21:40:35.537 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.530+0000 7f8fa5ffb640 10 client.?.objecter resend_mon_ops 2024-02-06T21:40:35.537 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fa5ffb640 1 -- 172.21.15.135:0/1397730557 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac0052d0 con 0x7f8fb00b54b0 2024-02-06T21:40:35.537 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fa5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:35.537 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fa5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:40:35.537 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fa5ffb640 10 monclient: dump: 2024-02-06T21:40:35.537 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:35.538 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:35.538 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.538 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.538 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:35.538 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.538 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:35.538 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:35.538 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:35.538 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.538 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fa5ffb640 10 monclient: _finish_auth 0 2024-02-06T21:40:35.538 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fa5ffb640 10 monclient: _check_auth_tickets 2024-02-06T21:40:35.538 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fa5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:35.538 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fa5ffb640 1 -- 172.21.15.135:0/1397730557 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fac005450 con 0x7f8fb00b54b0 2024-02-06T21:40:35.538 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fa5ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:35.539 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fa5ffb640 1 -- 172.21.15.135:0/1397730557 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac007260 con 0x7f8fb00b54b0 2024-02-06T21:40:35.539 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fb73a8640 5 monclient: authenticate success, global_id 4118 2024-02-06T21:40:35.539 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fb73a8640 10 monclient: _renew_subs 2024-02-06T21:40:35.539 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fb73a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:35.539 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fb73a8640 1 -- 172.21.15.135:0/1397730557 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb01333e0 con 0x7f8fb00b54b0 2024-02-06T21:40:35.539 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fa5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:35.539 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fa5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:40:35.539 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fa5ffb640 10 monclient: dump: 2024-02-06T21:40:35.539 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:35.539 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:35.539 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.539 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.539 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:35.539 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.540 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:35.540 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:35.540 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:35.540 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.540 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fa77fe640 4 set_mon_vals no callback set 2024-02-06T21:40:35.540 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fb73a8640 10 client.4118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:40:35.540 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fb73a8640 10 monclient: _renew_subs 2024-02-06T21:40:35.540 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fb73a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:35.540 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fb73a8640 1 -- 172.21.15.135:0/1397730557 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb0133130 con 0x7f8fb00b54b0 2024-02-06T21:40:35.540 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.534+0000 7f8fb73a8640 1 librados: init done 2024-02-06T21:40:35.540 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.535+0000 7f8fb73a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:35.540 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.535+0000 7f8fb73a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:35.540 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.535+0000 7f8fb73a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:35.540 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.535+0000 7f8fb73a8640 1 -- 172.21.15.135:0/1397730557 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fb0133620 con 0x7f8fb00b54b0 2024-02-06T21:40:35.575 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:35.574+0000 7f05e902c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-06T21:40:35.617 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:35.616+0000 7f05e902c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-06T21:40:35.635 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.634+0000 7f8fa5ffb640 1 -- 172.21.15.135:0/1397730557 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8fac0073c0 con 0x7f8fb00b54b0 2024-02-06T21:40:35.639 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.638+0000 7f8fa5ffb640 1 -- 172.21.15.135:0/1397730557 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f8fac003da0 con 0x7f8fb00b54b0 2024-02-06T21:40:35.639 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.638+0000 7f8fa5ffb640 10 client.4118.objecter ms_dispatch 0x7f8fb0000b90 osd_map(5..5 src has 1..5) v4 2024-02-06T21:40:35.640 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.638+0000 7f8fa5ffb640 3 client.4118.objecter handle_osd_map got epochs [5,5] > 0 2024-02-06T21:40:35.640 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.638+0000 7f8fa5ffb640 3 client.4118.objecter handle_osd_map decoding full epoch 5 2024-02-06T21:40:35.640 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.638+0000 7f8fa5ffb640 20 client.4118.objecter dump_active .. 0 homeless 2024-02-06T21:40:35.640 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.638+0000 7f8fa5ffb640 1 -- 172.21.15.135:0/1397730557 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f8fac020060 con 0x7f8fb00b54b0 2024-02-06T21:40:35.640 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.638+0000 7f8fa5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:35.640 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.638+0000 7f8fa5ffb640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:40:35.652 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:35.651+0000 7f4683038040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-06T21:40:35.697 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:35.696+0000 7f4683038040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-06T21:40:35.714 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.713+0000 7f8fb73a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "53534db0-2d4d-44ed-90e8-d67ea878b51a", "id": 3}] 2024-02-06T21:40:35.714 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.713+0000 7f8fb73a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "53534db0-2d4d-44ed-90e8-d67ea878b51a", "id": 3}] 2024-02-06T21:40:35.714 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.713+0000 7f8fb73a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:35.714 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.713+0000 7f8fb73a8640 1 -- 172.21.15.135:0/1397730557 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "53534db0-2d4d-44ed-90e8-d67ea878b51a", "id": 3} v 0) v1 -- 0x7f8fb004ef90 con 0x7f8fb00b54b0 2024-02-06T21:40:35.776 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:35.776+0000 7f05e902c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-06T21:40:35.779 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.778+0000 7f8fa5ffb640 1 -- 172.21.15.135:0/1397730557 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "53534db0-2d4d-44ed-90e8-d67ea878b51a", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8fac0038c0 con 0x7f8fb00b54b0 2024-02-06T21:40:35.779 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.779+0000 7f8fa5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "53534db0-2d4d-44ed-90e8-d67ea878b51a", "id": 3}] 2024-02-06T21:40:35.779 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.779+0000 7f8fa5ffb640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:40:35.780 INFO:teuthology.orchestra.run.smithi135.stdout:3 2024-02-06T21:40:35.781 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 10 librados: watch_flush enter 2024-02-06T21:40:35.781 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 10 librados: watch_flush exit 2024-02-06T21:40:35.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 20 client.4118.objecter shutdown clearing up homeless session... 2024-02-06T21:40:35.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 10 client.4118.objecter successfully canceled tick 2024-02-06T21:40:35.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 10 monclient: shutdown 2024-02-06T21:40:35.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:35.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 1 -- 172.21.15.135:0/1397730557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fb00b54b0 msgr2=0x7f8fb0138dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:35.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 1 --2- 172.21.15.135:0/1397730557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fb00b54b0 0x7f8fb0138dd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8fac00c340 tx=0x7f8fac00c800 comp rx=0 tx=0).stop 2024-02-06T21:40:35.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 1 -- 172.21.15.135:0/1397730557 shutdown_connections 2024-02-06T21:40:35.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 1 --2- 172.21.15.135:0/1397730557 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8fb013ff40 0x7f8fb01351c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 1 --2- 172.21.15.135:0/1397730557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fb0142680 0x7f8fb0135700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 1 --2- 172.21.15.135:0/1397730557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fb00b54b0 0x7f8fb0138dd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 1 -- 172.21.15.135:0/1397730557 >> 172.21.15.135:0/1397730557 conn(0x7f8fb00b8660 msgr2=0x7f8fb013f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:35.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 1 -- 172.21.15.135:0/1397730557 shutdown_connections 2024-02-06T21:40:35.783 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 1 -- 172.21.15.135:0/1397730557 wait complete. 2024-02-06T21:40:35.783 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.780+0000 7f8fb73a8640 1 librados: shutdown 2024-02-06T21:40:35.791 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new e4205ab5-787c-44e0-9b96-273b8c9cf1ee 4 2024-02-06T21:40:35.858 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:35.856+0000 7f4683038040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-06T21:40:35.890 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.889+0000 7f19f2379640 10 monclient: get_monmap_and_config 2024-02-06T21:40:35.891 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.889+0000 7f19f2379640 10 monclient: build_initial_monmap 2024-02-06T21:40:35.891 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.889+0000 7f19f2379640 10 monclient: monmap: 2024-02-06T21:40:35.891 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:35.891 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:35.891 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:40:35.890295+0000 2024-02-06T21:40:35.891 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:40:35.890295+0000 2024-02-06T21:40:35.891 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:35.891 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.891 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:40:35.891 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:40:35.891 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:40:35.891 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:40:35.892 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:40:35.892 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.894 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.890+0000 7f19f2379640 1 Processor -- start 2024-02-06T21:40:35.894 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.890+0000 7f19f2379640 1 -- start start 2024-02-06T21:40:35.894 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.890+0000 7f19f2379640 10 monclient: init 2024-02-06T21:40:35.894 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.891+0000 7f19f2379640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:35.895 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.891+0000 7f19f2379640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-02-06T21:40:35.895 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.891+0000 7f19f2379640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19ec0b54b0 addr v1:172.21.15.120:6789/0 2024-02-06T21:40:35.895 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.891+0000 7f19f2379640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f19ec13ff40 0x7f19ec0a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:35.895 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.891+0000 7f19f2379640 10 monclient(hunting): picked mon.noname-c con 0x7f19ec13ff40 addr v2:172.21.15.135:3300/0 2024-02-06T21:40:35.895 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.891+0000 7f19f2379640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19ec142680 addr v1:172.21.15.135:6789/0 2024-02-06T21:40:35.895 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.891+0000 7f19f2379640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec143200 con 0x7f19ec0b54b0 2024-02-06T21:40:35.895 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.891+0000 7f19f2379640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec0ab720 con 0x7f19ec142680 2024-02-06T21:40:35.895 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.891+0000 7f19f2379640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:35.895 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.891+0000 7f19f2379640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ec0ab520 con 0x7f19ec13ff40 2024-02-06T21:40:35.895 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.891+0000 7f19f2379640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:35.895 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.891+0000 7f19f2379640 10 monclient(hunting): authenticate will time out at 1277.213102s 2024-02-06T21:40:35.896 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.891+0000 7f19eb7fe640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f19ec13ff40 0x7f19ec0a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:35.896 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.891+0000 7f19ebfff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f19ec0b54b0 0x7f19ec13f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.135:40102/0 (socket says 172.21.15.135:40102) 2024-02-06T21:40:35.896 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.891+0000 7f19ebfff640 1 -- 172.21.15.135:0/3245958186 learned_addr learned my addr 172.21.15.135:0/3245958186 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-02-06T21:40:35.896 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.892+0000 7f19eb7fe640 10 monclient(hunting): get_auth_request con 0x7f19ec13ff40 auth_method 0 2024-02-06T21:40:35.896 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.892+0000 7f19eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:35.896 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.892+0000 7f19eb7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:35.896 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.892+0000 7f19eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:35.896 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.893+0000 7f19eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:35.896 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.893+0000 7f19eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:35.896 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.893+0000 7f19eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:35.897 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.894+0000 7f19eaffd640 1 -- 172.21.15.135:0/3245958186 <== mon.0 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182745883 0 0) 0x7f19ec143200 con 0x7f19ec0b54b0 2024-02-06T21:40:35.897 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.894+0000 7f19eaffd640 10 monclient(hunting): _negotiate 2024-02-06T21:40:35.898 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.894+0000 7f19eaffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:35.898 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.894+0000 7f19eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:35.898 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.894+0000 7f19eaffd640 10 monclient(hunting): my global_id is 4134 2024-02-06T21:40:35.898 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.894+0000 7f19eaffd640 1 -- 172.21.15.135:0/3245958186 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d4003520 con 0x7f19ec0b54b0 2024-02-06T21:40:35.898 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.895+0000 7f19eb7fe640 10 monclient(hunting): handle_auth_done global_id 4132 payload 274 2024-02-06T21:40:35.898 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.895+0000 7f19eb7fe640 1 -- 172.21.15.135:0/3245958186 >> v1:172.21.15.135:6789/0 conn(0x7f19ec142680 legacy=0x7f19ec142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:35.898 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.895+0000 7f19eb7fe640 1 -- 172.21.15.135:0/3245958186 >> v1:172.21.15.120:6789/0 conn(0x7f19ec0b54b0 legacy=0x7f19ec13f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:35.898 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.895+0000 7f19eb7fe640 10 monclient: _finish_hunting 0 2024-02-06T21:40:35.898 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.895+0000 7f19eb7fe640 1 monclient: found mon.noname-c 2024-02-06T21:40:35.898 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.895+0000 7f19eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:35.899 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.895+0000 7f19eb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T21:40:35.899 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.895+0000 7f19eb7fe640 1 -- 172.21.15.135:0/3245958186 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec0b4800 con 0x7f19ec13ff40 2024-02-06T21:40:35.900 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.899+0000 7f19eb7fe640 1 --2- 172.21.15.135:0/3245958186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19ec13ff40 0x7f19ec0a9860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0097a0 tx=0x7f19dc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a465a88f8aa92cb server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:35.905 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19eaffd640 1 -- 172.21.15.135:0/3245958186 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc030610 con 0x7f19ec13ff40 2024-02-06T21:40:35.905 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:35.905 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:40:35.905 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19eaffd640 10 monclient: dump: 2024-02-06T21:40:35.905 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:35.906 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:35.906 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.906 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.906 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:35.906 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.906 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:35.906 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:35.906 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:35.906 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.906 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19eaffd640 10 monclient: _finish_auth 0 2024-02-06T21:40:35.907 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19eaffd640 10 monclient: _check_auth_tickets 2024-02-06T21:40:35.907 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:35.907 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19eaffd640 1 -- 172.21.15.135:0/3245958186 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19dc030770 con 0x7f19ec13ff40 2024-02-06T21:40:35.907 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19f2379640 5 monclient: authenticate success, global_id 4132 2024-02-06T21:40:35.907 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19f2379640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:40:35.907 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19eaffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:35.907 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19eaffd640 1 -- 172.21.15.135:0/3245958186 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc030b00 con 0x7f19ec13ff40 2024-02-06T21:40:35.907 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19f2379640 10 monclient: get_monmap_and_config success 2024-02-06T21:40:35.907 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19f2379640 4 set_mon_vals no callback set 2024-02-06T21:40:35.907 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:35.907 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:40:35.907 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.903+0000 7f19eaffd640 10 monclient: dump: 2024-02-06T21:40:35.907 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:35.908 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:35.908 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.908 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.908 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:35.908 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.908 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:35.908 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:35.908 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:35.908 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.908 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.904+0000 7f19f2379640 10 monclient: shutdown 2024-02-06T21:40:35.908 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.904+0000 7f19f2379640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:35.909 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.904+0000 7f19f2379640 1 -- 172.21.15.135:0/3245958186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19ec13ff40 msgr2=0x7f19ec0a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:35.909 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.904+0000 7f19f2379640 1 --2- 172.21.15.135:0/3245958186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19ec13ff40 0x7f19ec0a9860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0097a0 tx=0x7f19dc02b960 comp rx=0 tx=0).stop 2024-02-06T21:40:35.909 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.904+0000 7f19f2379640 1 -- 172.21.15.135:0/3245958186 shutdown_connections 2024-02-06T21:40:35.909 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.904+0000 7f19f2379640 1 --2- 172.21.15.135:0/3245958186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19ec13ff40 0x7f19ec0a9860 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.909 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.904+0000 7f19f2379640 1 -- 172.21.15.135:0/3245958186 >> 172.21.15.135:0/3245958186 conn(0x7f19ec0b8660 msgr2=0x7f19ec0b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:35.909 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.904+0000 7f19f2379640 1 -- 172.21.15.135:0/3245958186 shutdown_connections 2024-02-06T21:40:35.909 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.904+0000 7f19f2379640 1 -- 172.21.15.135:0/3245958186 wait complete. 2024-02-06T21:40:35.909 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.904+0000 7f19f2379640 10 monclient: build_initial_monmap 2024-02-06T21:40:35.909 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.904+0000 7f19f2379640 10 monclient: monmap: 2024-02-06T21:40:35.909 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:35.909 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:40:35.910 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2024-02-06T21:40:35.910 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2024-02-06T21:40:35.910 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:35.910 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.910 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:40:35.910 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:40:35.910 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:40:35.910 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.914 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.905+0000 7f19f2379640 1 librados: starting msgr at 2024-02-06T21:40:35.914 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.905+0000 7f19f2379640 1 librados: starting objecter 2024-02-06T21:40:35.914 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.905+0000 7f19f2379640 1 Processor -- start 2024-02-06T21:40:35.914 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.905+0000 7f19f2379640 1 -- start start 2024-02-06T21:40:35.914 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.905+0000 7f19f2379640 1 librados: setting wanted keys 2024-02-06T21:40:35.914 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.905+0000 7f19f2379640 1 librados: calling monclient init 2024-02-06T21:40:35.914 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.905+0000 7f19f2379640 10 monclient: init 2024-02-06T21:40:35.914 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.905+0000 7f19f2379640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:35.914 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.905+0000 7f19f2379640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T21:40:35.914 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.905+0000 7f19f2379640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19ec0b54b0 0x7f19ec138530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:35.915 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19f2379640 10 monclient(hunting): picked mon.noname-b con 0x7f19ec0b54b0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:40:35.915 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19f2379640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19ec13ff40 0x7f19ec0ac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:35.915 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19f2379640 10 monclient(hunting): picked mon.noname-a con 0x7f19ec13ff40 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:40:35.915 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19f2379640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19ec142680 0x7f19ec135090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:35.915 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19f2379640 10 monclient(hunting): picked mon.noname-c con 0x7f19ec142680 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:35.915 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19f2379640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:35.915 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19f2379640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec138b60 con 0x7f19ec13ff40 2024-02-06T21:40:35.915 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19f2379640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19f2379640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0abed0 con 0x7f19ec0b54b0 2024-02-06T21:40:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19f2379640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19f2379640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0ac1b0 con 0x7f19ec142680 2024-02-06T21:40:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19f2379640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19f2379640 10 monclient(hunting): authenticate will time out at 1277.227999s 2024-02-06T21:40:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19ebfff640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19ec0b54b0 0x7f19ec138530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19eb7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19ec13ff40 0x7f19ec0ac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19ebfff640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19ec0b54b0 0x7f19ec138530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:47478/0 (socket says 172.21.15.135:47478) 2024-02-06T21:40:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19f0916640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19ec142680 0x7f19ec135090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19ebfff640 1 -- 172.21.15.135:0/1353034447 learned_addr learned my addr 172.21.15.135:0/1353034447 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-02-06T21:40:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19ebfff640 10 monclient(hunting): get_auth_request con 0x7f19ec0b54b0 auth_method 0 2024-02-06T21:40:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19ebfff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.906+0000 7f19ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19f0916640 10 monclient(hunting): get_auth_request con 0x7f19ec142680 auth_method 0 2024-02-06T21:40:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19f0916640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19f0916640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19f0916640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19eb7fe640 10 monclient(hunting): get_auth_request con 0x7f19ec13ff40 auth_method 0 2024-02-06T21:40:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19eb7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:35.918 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:35.918 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19f0916640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:35.918 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19f0916640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:35.918 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19f0916640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:35.918 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:35.918 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:35.918 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.907+0000 7f19eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:35.918 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.910+0000 7f19ebfff640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-02-06T21:40:35.918 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.910+0000 7f19ebfff640 1 -- 172.21.15.135:0/1353034447 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19ec142680 msgr2=0x7f19ec135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:35.918 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.910+0000 7f19ebfff640 1 --2- 172.21.15.135:0/1353034447 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19ec142680 0x7f19ec135090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.918 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.910+0000 7f19ebfff640 1 -- 172.21.15.135:0/1353034447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19ec13ff40 msgr2=0x7f19ec0ac5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:35.918 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.910+0000 7f19ebfff640 1 --2- 172.21.15.135:0/1353034447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19ec13ff40 0x7f19ec0ac5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:35.919 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.910+0000 7f19ebfff640 10 monclient: _finish_hunting 0 2024-02-06T21:40:35.919 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.910+0000 7f19ebfff640 1 monclient: found mon.noname-b 2024-02-06T21:40:35.919 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.910+0000 7f19ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:35.919 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.910+0000 7f19ebfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:40:35.919 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.910+0000 7f19ebfff640 1 -- 172.21.15.135:0/1353034447 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc009400 con 0x7f19ec0b54b0 2024-02-06T21:40:35.919 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.912+0000 7f19ebfff640 1 --2- 172.21.15.135:0/1353034447 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19ec0b54b0 0x7f19ec138530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f19e000c410 tx=0x7f19e000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:35.919 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.912+0000 7f19e8ff9640 10 client.?.objecter ms_handle_connect 0x7f19ec0b54b0 2024-02-06T21:40:35.919 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.912+0000 7f19e8ff9640 10 client.?.objecter resend_mon_ops 2024-02-06T21:40:35.921 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.915+0000 7f19e8ff9640 1 -- 172.21.15.135:0/1353034447 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0005ce0 con 0x7f19ec0b54b0 2024-02-06T21:40:35.921 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.915+0000 7f19e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:35.921 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19e8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:40:35.921 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19e8ff9640 10 monclient: dump: 2024-02-06T21:40:35.921 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:35.921 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:35.921 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.921 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.922 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:35.922 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.922 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:35.922 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:35.922 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:35.922 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.922 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19e8ff9640 10 monclient: _finish_auth 0 2024-02-06T21:40:35.922 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19e8ff9640 10 monclient: _check_auth_tickets 2024-02-06T21:40:35.922 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:35.922 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19e8ff9640 1 -- 172.21.15.135:0/1353034447 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e0005e40 con 0x7f19ec0b54b0 2024-02-06T21:40:35.922 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19f2379640 5 monclient: authenticate success, global_id 4138 2024-02-06T21:40:35.922 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19f2379640 10 monclient: _renew_subs 2024-02-06T21:40:35.922 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19f2379640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:35.923 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19f2379640 1 -- 172.21.15.135:0/1353034447 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ec1396c0 con 0x7f19ec0b54b0 2024-02-06T21:40:35.923 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:35.923 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19e8ff9640 1 -- 172.21.15.135:0/1353034447 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e00072f0 con 0x7f19ec0b54b0 2024-02-06T21:40:35.923 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:35.923 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19e8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:40:35.923 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19e8ff9640 10 monclient: dump: 2024-02-06T21:40:35.923 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:35.923 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:35.923 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.923 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:35.923 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:35.923 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:35.923 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:35.923 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:35.924 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:35.924 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:35.924 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19ea7fc640 4 set_mon_vals no callback set 2024-02-06T21:40:35.924 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19f2379640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:40:35.924 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19f2379640 10 monclient: _renew_subs 2024-02-06T21:40:35.924 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19f2379640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:35.924 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19f2379640 1 -- 172.21.15.135:0/1353034447 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ec137890 con 0x7f19ec0b54b0 2024-02-06T21:40:35.924 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.916+0000 7f19f2379640 1 librados: init done 2024-02-06T21:40:35.924 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.917+0000 7f19f2379640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:35.924 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.917+0000 7f19f2379640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:35.924 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.917+0000 7f19f2379640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:35.925 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.917+0000 7f19f2379640 1 -- 172.21.15.135:0/1353034447 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19ec04ef90 con 0x7f19ec0b54b0 2024-02-06T21:40:35.925 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.918+0000 7f19e8ff9640 1 -- 172.21.15.135:0/1353034447 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f19e0007510 con 0x7f19ec0b54b0 2024-02-06T21:40:35.925 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.918+0000 7f19e8ff9640 1 -- 172.21.15.135:0/1353034447 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f19e001b8c0 con 0x7f19ec0b54b0 2024-02-06T21:40:35.925 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.918+0000 7f19e8ff9640 10 client.4138.objecter ms_dispatch 0x7f19ec000b90 osd_map(6..6 src has 1..6) v4 2024-02-06T21:40:35.925 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.918+0000 7f19e8ff9640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-02-06T21:40:35.925 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.918+0000 7f19e8ff9640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-02-06T21:40:35.925 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:35.918+0000 7f19e8ff9640 20 client.4138.objecter dump_active .. 0 homeless 2024-02-06T21:40:36.017 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.016+0000 7f19e8ff9640 1 -- 172.21.15.135:0/1353034447 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f19e0017070 con 0x7f19ec0b54b0 2024-02-06T21:40:36.018 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.016+0000 7f19e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:36.018 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.016+0000 7f19e8ff9640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:40:36.090 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.089+0000 7f19f2379640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e4205ab5-787c-44e0-9b96-273b8c9cf1ee", "id": 4}] 2024-02-06T21:40:36.090 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.089+0000 7f19f2379640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e4205ab5-787c-44e0-9b96-273b8c9cf1ee", "id": 4}] 2024-02-06T21:40:36.090 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.089+0000 7f19f2379640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:36.090 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.089+0000 7f19f2379640 1 -- 172.21.15.135:0/1353034447 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e4205ab5-787c-44e0-9b96-273b8c9cf1ee", "id": 4} v 0) v1 -- 0x7f19ec067e10 con 0x7f19ec0b54b0 2024-02-06T21:40:36.166 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.165+0000 7f19e8ff9640 1 -- 172.21.15.135:0/1353034447 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e4205ab5-787c-44e0-9b96-273b8c9cf1ee", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f19e001b4e0 con 0x7f19ec0b54b0 2024-02-06T21:40:36.166 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.165+0000 7f19e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e4205ab5-787c-44e0-9b96-273b8c9cf1ee", "id": 4}] 2024-02-06T21:40:36.166 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.165+0000 7f19e8ff9640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:40:36.166 INFO:teuthology.orchestra.run.smithi135.stdout:4 2024-02-06T21:40:36.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 10 librados: watch_flush enter 2024-02-06T21:40:36.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 10 librados: watch_flush exit 2024-02-06T21:40:36.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 20 client.4138.objecter shutdown clearing up homeless session... 2024-02-06T21:40:36.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 10 client.4138.objecter successfully canceled tick 2024-02-06T21:40:36.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 10 monclient: shutdown 2024-02-06T21:40:36.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:36.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 1 -- 172.21.15.135:0/1353034447 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19ec0b54b0 msgr2=0x7f19ec138530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:36.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 1 --2- 172.21.15.135:0/1353034447 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19ec0b54b0 0x7f19ec138530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f19e000c410 tx=0x7f19e000c8d0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 1 -- 172.21.15.135:0/1353034447 shutdown_connections 2024-02-06T21:40:36.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 1 --2- 172.21.15.135:0/1353034447 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19ec142680 0x7f19ec135090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 1 --2- 172.21.15.135:0/1353034447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19ec13ff40 0x7f19ec0ac5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.168 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 1 --2- 172.21.15.135:0/1353034447 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19ec0b54b0 0x7f19ec138530 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.169 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 1 -- 172.21.15.135:0/1353034447 >> 172.21.15.135:0/1353034447 conn(0x7f19ec0b8660 msgr2=0x7f19ec13c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:36.169 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 1 -- 172.21.15.135:0/1353034447 shutdown_connections 2024-02-06T21:40:36.169 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 1 -- 172.21.15.135:0/1353034447 wait complete. 2024-02-06T21:40:36.169 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.166+0000 7f19f2379640 1 librados: shutdown 2024-02-06T21:40:36.180 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 43b600bc-3afe-44bd-bd3f-690106e7e0ce 5 2024-02-06T21:40:36.286 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.284+0000 7f089db04640 10 monclient: get_monmap_and_config 2024-02-06T21:40:36.286 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.284+0000 7f089db04640 10 monclient: build_initial_monmap 2024-02-06T21:40:36.286 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.284+0000 7f089db04640 10 monclient: monmap: 2024-02-06T21:40:36.286 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:36.286 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:36.286 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:40:36.285162+0000 2024-02-06T21:40:36.286 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:40:36.285162+0000 2024-02-06T21:40:36.286 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:36.286 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:36.286 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:40:36.287 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:40:36.287 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:40:36.287 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:40:36.287 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:40:36.287 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:36.287 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 1 Processor -- start 2024-02-06T21:40:36.287 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 1 -- start start 2024-02-06T21:40:36.287 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 10 monclient: init 2024-02-06T21:40:36.289 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:36.289 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-02-06T21:40:36.289 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08980b54b0 addr v1:172.21.15.120:6789/0 2024-02-06T21:40:36.289 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f089813ff40 0x7f08980a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:36.289 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 10 monclient(hunting): picked mon.noname-a con 0x7f089813ff40 addr v2:172.21.15.120:3300/0 2024-02-06T21:40:36.289 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0898142680 0x7f0898142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:36.289 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 10 monclient(hunting): picked mon.noname-b con 0x7f0898142680 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:36.289 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08980ab720 con 0x7f08980b54b0 2024-02-06T21:40:36.289 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:36.290 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08980ab520 con 0x7f089813ff40 2024-02-06T21:40:36.290 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:36.290 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08980b5890 con 0x7f0898142680 2024-02-06T21:40:36.290 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:36.290 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.285+0000 7f089db04640 10 monclient(hunting): authenticate will time out at 1277.607857s 2024-02-06T21:40:36.290 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.286+0000 7f0896ffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f089813ff40 0x7f08980a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:36.290 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.286+0000 7f0897fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0898142680 0x7f0898142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:36.290 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.286+0000 7f0896ffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f089813ff40 0x7f08980a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.135:37128/0 (socket says 172.21.15.135:37128) 2024-02-06T21:40:36.290 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.286+0000 7f0896ffd640 1 -- 172.21.15.135:0/2697175503 learned_addr learned my addr 172.21.15.135:0/2697175503 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-02-06T21:40:36.290 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.286+0000 7f0896ffd640 10 monclient(hunting): get_auth_request con 0x7f089813ff40 auth_method 0 2024-02-06T21:40:36.290 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.286+0000 7f0896ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:36.291 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.286+0000 7f0896ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:36.291 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.286+0000 7f0896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:36.291 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.287+0000 7f0897fff640 10 monclient(hunting): get_auth_request con 0x7f0898142680 auth_method 0 2024-02-06T21:40:36.291 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.287+0000 7f0897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:36.291 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.287+0000 7f0897fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:36.291 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.287+0000 7f0897fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:36.291 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.287+0000 7f0896ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:36.291 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.287+0000 7f0896ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:36.291 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.287+0000 7f0896ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:36.291 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.287+0000 7f0897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:36.291 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.287+0000 7f0897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:36.291 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.287+0000 7f0897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:36.292 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.289+0000 7f0897fff640 10 monclient(hunting): handle_auth_done global_id 4124 payload 274 2024-02-06T21:40:36.292 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.289+0000 7f0897fff640 1 -- 172.21.15.135:0/2697175503 >> v2:172.21.15.120:3300/0 conn(0x7f089813ff40 msgr2=0x7f08980a9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:36.292 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.289+0000 7f0897fff640 1 --2- 172.21.15.135:0/2697175503 >> v2:172.21.15.120:3300/0 conn(0x7f089813ff40 0x7f08980a9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.292 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.289+0000 7f0897fff640 1 -- 172.21.15.135:0/2697175503 >> v1:172.21.15.120:6789/0 conn(0x7f08980b54b0 legacy=0x7f089813f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:36.292 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.289+0000 7f0897fff640 10 monclient: _finish_hunting 0 2024-02-06T21:40:36.292 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.289+0000 7f0897fff640 1 monclient: found mon.noname-b 2024-02-06T21:40:36.292 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.289+0000 7f0897fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:36.293 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.289+0000 7f0897fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:40:36.293 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.289+0000 7f0897fff640 1 -- 172.21.15.135:0/2697175503 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08980b4800 con 0x7f0898142680 2024-02-06T21:40:36.293 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.289+0000 7f0896ffd640 1 --2- 172.21.15.135:0/2697175503 >> v2:172.21.15.120:3300/0 conn(0x7f089813ff40 0x7f08980a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-06T21:40:36.293 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.291+0000 7f0897fff640 1 --2- 172.21.15.135:0/2697175503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0898142680 0x7f0898142ac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0890009d90 tx=0x7f089002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df0bbb66fa4d560d server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:36.296 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.294+0000 7f08967fc640 1 -- 172.21.15.135:0/2697175503 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089002cbc0 con 0x7f0898142680 2024-02-06T21:40:36.296 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.294+0000 7f08967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:36.296 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f08967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:40:36.296 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f08967fc640 10 monclient: dump: 2024-02-06T21:40:36.296 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:36.296 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:36.296 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.296 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.296 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:36.296 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:36.296 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:36.296 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:36.296 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:36.296 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f08967fc640 10 monclient: _finish_auth 0 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f08967fc640 10 monclient: _check_auth_tickets 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f08967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f08967fc640 1 -- 172.21.15.135:0/2697175503 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089002cd20 con 0x7f0898142680 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f08967fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f08967fc640 1 -- 172.21.15.135:0/2697175503 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890034660 con 0x7f0898142680 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f089db04640 5 monclient: authenticate success, global_id 4124 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f089db04640 10 monclient: get_monmap_and_config success 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f089db04640 4 set_mon_vals no callback set 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f08967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f08967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f08967fc640 10 monclient: dump: 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.297 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.298 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:36.298 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:36.298 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:36.298 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:36.298 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:36.298 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:36.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f089db04640 10 monclient: shutdown 2024-02-06T21:40:36.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f089db04640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:36.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f089db04640 1 -- 172.21.15.135:0/2697175503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0898142680 msgr2=0x7f0898142ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:36.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f089db04640 1 --2- 172.21.15.135:0/2697175503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0898142680 0x7f0898142ac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0890009d90 tx=0x7f089002be10 comp rx=0 tx=0).stop 2024-02-06T21:40:36.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f089db04640 1 -- 172.21.15.135:0/2697175503 shutdown_connections 2024-02-06T21:40:36.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f089db04640 1 --2- 172.21.15.135:0/2697175503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0898142680 0x7f0898142ac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f089db04640 1 --2- 172.21.15.135:0/2697175503 >> v2:172.21.15.120:3300/0 conn(0x7f089813ff40 0x7f08980a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f089db04640 1 -- 172.21.15.135:0/2697175503 >> 172.21.15.135:0/2697175503 conn(0x7f08980b8660 msgr2=0x7f08980b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:36.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f089db04640 1 -- 172.21.15.135:0/2697175503 shutdown_connections 2024-02-06T21:40:36.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.295+0000 7f089db04640 1 -- 172.21.15.135:0/2697175503 wait complete. 2024-02-06T21:40:36.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.296+0000 7f089db04640 10 monclient: build_initial_monmap 2024-02-06T21:40:36.304 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.296+0000 7f089db04640 10 monclient: monmap: 2024-02-06T21:40:36.304 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:36.304 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:40:36.304 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2024-02-06T21:40:36.304 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2024-02-06T21:40:36.304 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:36.304 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:36.304 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:40:36.304 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:40:36.304 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:40:36.304 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:36.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.296+0000 7f089db04640 1 librados: starting msgr at 2024-02-06T21:40:36.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.296+0000 7f089db04640 1 librados: starting objecter 2024-02-06T21:40:36.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.296+0000 7f089db04640 1 Processor -- start 2024-02-06T21:40:36.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.296+0000 7f089db04640 1 -- start start 2024-02-06T21:40:36.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.296+0000 7f089db04640 1 librados: setting wanted keys 2024-02-06T21:40:36.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.296+0000 7f089db04640 1 librados: calling monclient init 2024-02-06T21:40:36.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.296+0000 7f089db04640 10 monclient: init 2024-02-06T21:40:36.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:36.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 10 monclient: _add_conns ranks=[2,0,1] 2024-02-06T21:40:36.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08980b54b0 0x7f0898138b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:36.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 10 monclient(hunting): picked mon.noname-c con 0x7f08980b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:36.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f089813ff40 0x7f0898136640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:36.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 10 monclient(hunting): picked mon.noname-a con 0x7f089813ff40 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:40:36.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0898142680 0x7f0898131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:36.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 10 monclient(hunting): picked mon.noname-b con 0x7f0898142680 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:40:36.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:36.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0898137e50 con 0x7f089813ff40 2024-02-06T21:40:36.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:36.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0898137390 con 0x7f0898142680 2024-02-06T21:40:36.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:36.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08981380a0 con 0x7f08980b54b0 2024-02-06T21:40:36.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:36.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f089db04640 10 monclient(hunting): authenticate will time out at 1277.619388s 2024-02-06T21:40:36.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.297+0000 7f08977fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08980b54b0 0x7f0898138b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:36.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0897fff640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0898142680 0x7f0898131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:36.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0896ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f089813ff40 0x7f0898136640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:36.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f08977fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08980b54b0 0x7f0898138b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.135:37158/0 (socket says 172.21.15.135:37158) 2024-02-06T21:40:36.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f08977fe640 1 -- 172.21.15.135:0/3541645642 learned_addr learned my addr 172.21.15.135:0/3541645642 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-02-06T21:40:36.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f08977fe640 10 monclient(hunting): get_auth_request con 0x7f08980b54b0 auth_method 0 2024-02-06T21:40:36.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f08977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:36.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f08977fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:36.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f08977fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:36.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0897fff640 10 monclient(hunting): get_auth_request con 0x7f0898142680 auth_method 0 2024-02-06T21:40:36.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:36.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0897fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:36.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0897fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:36.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0896ffd640 10 monclient(hunting): get_auth_request con 0x7f089813ff40 auth_method 0 2024-02-06T21:40:36.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0896ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:36.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0896ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:36.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:36.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f08977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:36.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f08977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:36.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f08977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:36.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0896ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:36.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0896ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:36.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0896ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:36.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:36.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:36.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.298+0000 7f0897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:36.310 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.301+0000 7f08977fe640 10 monclient(hunting): handle_auth_done global_id 4127 payload 824 2024-02-06T21:40:36.310 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.301+0000 7f08977fe640 1 -- 172.21.15.135:0/3541645642 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0898142680 msgr2=0x7f0898131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:36.310 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.301+0000 7f08977fe640 1 --2- 172.21.15.135:0/3541645642 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0898142680 0x7f0898131d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.310 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.301+0000 7f08977fe640 1 -- 172.21.15.135:0/3541645642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f089813ff40 msgr2=0x7f0898136640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:36.310 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.301+0000 7f08977fe640 1 --2- 172.21.15.135:0/3541645642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f089813ff40 0x7f0898136640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.301+0000 7f08977fe640 10 monclient: _finish_hunting 0 2024-02-06T21:40:36.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.301+0000 7f08977fe640 1 monclient: found mon.noname-c 2024-02-06T21:40:36.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.301+0000 7f08977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:36.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.301+0000 7f08977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:40:36.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.301+0000 7f08977fe640 1 -- 172.21.15.135:0/3541645642 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c0094d0 con 0x7f08980b54b0 2024-02-06T21:40:36.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.303+0000 7f08977fe640 1 --2- 172.21.15.135:0/3541645642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08980b54b0 0x7f0898138b10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f088800c440 tx=0x7f088800c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:36.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.303+0000 7f087ffff640 10 client.?.objecter ms_handle_connect 0x7f08980b54b0 2024-02-06T21:40:36.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.303+0000 7f087ffff640 10 client.?.objecter resend_mon_ops 2024-02-06T21:40:36.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f087ffff640 1 -- 172.21.15.135:0/3541645642 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0888005480 con 0x7f08980b54b0 2024-02-06T21:40:36.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f087ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:36.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f087ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:40:36.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f087ffff640 10 monclient: dump: 2024-02-06T21:40:36.311 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:36.311 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:36.312 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.312 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.312 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:36.312 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:36.312 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:36.312 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:36.312 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:36.312 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:36.312 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f087ffff640 10 monclient: _finish_auth 0 2024-02-06T21:40:36.312 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f087ffff640 10 monclient: _check_auth_tickets 2024-02-06T21:40:36.312 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f087ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:36.312 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f087ffff640 1 -- 172.21.15.135:0/3541645642 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08880035f0 con 0x7f08980b54b0 2024-02-06T21:40:36.312 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f089db04640 5 monclient: authenticate success, global_id 4127 2024-02-06T21:40:36.312 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f089db04640 10 monclient: _renew_subs 2024-02-06T21:40:36.312 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f089db04640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f089db04640 1 -- 172.21.15.135:0/3541645642 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0890009a70 con 0x7f08980b54b0 2024-02-06T21:40:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f087ffff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f087ffff640 1 -- 172.21.15.135:0/3541645642 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08880039c0 con 0x7f08980b54b0 2024-02-06T21:40:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f087ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f087ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:40:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f087ffff640 10 monclient: dump: 2024-02-06T21:40:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:36.314 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:36.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f0895ffb640 4 set_mon_vals no callback set 2024-02-06T21:40:36.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f089db04640 10 client.4127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:40:36.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f089db04640 10 monclient: _renew_subs 2024-02-06T21:40:36.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f089db04640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:36.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f089db04640 1 -- 172.21.15.135:0/3541645642 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0898138430 con 0x7f08980b54b0 2024-02-06T21:40:36.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.307+0000 7f089db04640 1 librados: init done 2024-02-06T21:40:36.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.308+0000 7f089db04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:36.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.308+0000 7f089db04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:36.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.308+0000 7f089db04640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:36.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.308+0000 7f089db04640 1 -- 172.21.15.135:0/3541645642 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f089804ef90 con 0x7f08980b54b0 2024-02-06T21:40:36.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.310+0000 7f087ffff640 1 -- 172.21.15.135:0/3541645642 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0888003b20 con 0x7f08980b54b0 2024-02-06T21:40:36.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.310+0000 7f087ffff640 1 -- 172.21.15.135:0/3541645642 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f088801b930 con 0x7f08980b54b0 2024-02-06T21:40:36.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.310+0000 7f087ffff640 10 client.4127.objecter ms_dispatch 0x7f0898000b90 osd_map(7..7 src has 1..7) v4 2024-02-06T21:40:36.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.310+0000 7f087ffff640 3 client.4127.objecter handle_osd_map got epochs [7,7] > 0 2024-02-06T21:40:36.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.310+0000 7f087ffff640 3 client.4127.objecter handle_osd_map decoding full epoch 7 2024-02-06T21:40:36.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.310+0000 7f087ffff640 20 client.4127.objecter dump_active .. 0 homeless 2024-02-06T21:40:36.408 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.407+0000 7f087ffff640 1 -- 172.21.15.135:0/3541645642 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0888017070 con 0x7f08980b54b0 2024-02-06T21:40:36.408 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.407+0000 7f087ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:36.408 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.407+0000 7f087ffff640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:40:36.479 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.478+0000 7f089db04640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "43b600bc-3afe-44bd-bd3f-690106e7e0ce", "id": 5}] 2024-02-06T21:40:36.479 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.478+0000 7f089db04640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "43b600bc-3afe-44bd-bd3f-690106e7e0ce", "id": 5}] 2024-02-06T21:40:36.479 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.478+0000 7f089db04640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:36.479 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.478+0000 7f089db04640 1 -- 172.21.15.135:0/3541645642 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "43b600bc-3afe-44bd-bd3f-690106e7e0ce", "id": 5} v 0) v1 -- 0x7f08981374d0 con 0x7f08980b54b0 2024-02-06T21:40:36.542 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.541+0000 7f087ffff640 1 -- 172.21.15.135:0/3541645642 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "43b600bc-3afe-44bd-bd3f-690106e7e0ce", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f08981374d0 con 0x7f08980b54b0 2024-02-06T21:40:36.542 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.541+0000 7f087ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "43b600bc-3afe-44bd-bd3f-690106e7e0ce", "id": 5}] 2024-02-06T21:40:36.542 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.541+0000 7f087ffff640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:40:36.543 INFO:teuthology.orchestra.run.smithi135.stdout:5 2024-02-06T21:40:36.544 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 10 librados: watch_flush enter 2024-02-06T21:40:36.544 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 10 librados: watch_flush exit 2024-02-06T21:40:36.544 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 20 client.4127.objecter shutdown clearing up homeless session... 2024-02-06T21:40:36.544 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 10 client.4127.objecter successfully canceled tick 2024-02-06T21:40:36.544 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 10 monclient: shutdown 2024-02-06T21:40:36.544 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:36.545 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 1 -- 172.21.15.135:0/3541645642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08980b54b0 msgr2=0x7f0898138b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:36.545 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 1 --2- 172.21.15.135:0/3541645642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08980b54b0 0x7f0898138b10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f088800c440 tx=0x7f088800c900 comp rx=0 tx=0).stop 2024-02-06T21:40:36.545 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 1 -- 172.21.15.135:0/3541645642 shutdown_connections 2024-02-06T21:40:36.545 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 1 --2- 172.21.15.135:0/3541645642 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0898142680 0x7f0898131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.545 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 1 --2- 172.21.15.135:0/3541645642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f089813ff40 0x7f0898136640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.545 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 1 --2- 172.21.15.135:0/3541645642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08980b54b0 0x7f0898138b10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.545 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 1 -- 172.21.15.135:0/3541645642 >> 172.21.15.135:0/3541645642 conn(0x7f08980b8660 msgr2=0x7f089813f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:36.545 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 1 -- 172.21.15.135:0/3541645642 shutdown_connections 2024-02-06T21:40:36.545 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 1 -- 172.21.15.135:0/3541645642 wait complete. 2024-02-06T21:40:36.545 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.542+0000 7f089db04640 1 librados: shutdown 2024-02-06T21:40:36.553 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 1d6b9f96-1f9d-44f2-83b0-93f5c342fafb 6 2024-02-06T21:40:36.639 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:36.638+0000 7f05e902c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-06T21:40:36.654 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.653+0000 7ffa248e0640 10 monclient: get_monmap_and_config 2024-02-06T21:40:36.654 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.653+0000 7ffa248e0640 10 monclient: build_initial_monmap 2024-02-06T21:40:36.654 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.653+0000 7ffa248e0640 10 monclient: monmap: 2024-02-06T21:40:36.654 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:36.654 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:36.654 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:40:36.654299+0000 2024-02-06T21:40:36.655 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:40:36.654299+0000 2024-02-06T21:40:36.655 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:36.655 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:36.655 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:40:36.655 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:40:36.655 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:40:36.655 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:40:36.655 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:40:36.655 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:36.657 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.654+0000 7ffa248e0640 1 Processor -- start 2024-02-06T21:40:36.657 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.654+0000 7ffa248e0640 1 -- start start 2024-02-06T21:40:36.657 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.654+0000 7ffa248e0640 10 monclient: init 2024-02-06T21:40:36.657 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.654+0000 7ffa248e0640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:36.657 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.655+0000 7ffa248e0640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-02-06T21:40:36.658 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.655+0000 7ffa248e0640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffa200b54b0 0x7ffa200a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:36.658 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.655+0000 7ffa248e0640 10 monclient(hunting): picked mon.noname-a con 0x7ffa200b54b0 addr v2:172.21.15.120:3300/0 2024-02-06T21:40:36.658 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.655+0000 7ffa248e0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa2013f830 addr v1:172.21.15.120:6789/0 2024-02-06T21:40:36.658 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.655+0000 7ffa248e0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa20142680 0x7ffa20142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:36.658 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.655+0000 7ffa248e0640 10 monclient(hunting): picked mon.noname-b con 0x7ffa20142680 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:36.658 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.655+0000 7ffa248e0640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa200ab720 con 0x7ffa2013f830 2024-02-06T21:40:36.658 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.655+0000 7ffa248e0640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:36.658 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.655+0000 7ffa248e0640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa200ab520 con 0x7ffa200b54b0 2024-02-06T21:40:36.658 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.655+0000 7ffa248e0640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:36.658 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.655+0000 7ffa248e0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa200b5890 con 0x7ffa20142680 2024-02-06T21:40:36.658 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.655+0000 7ffa248e0640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:36.658 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.655+0000 7ffa248e0640 10 monclient(hunting): authenticate will time out at 1277.977056s 2024-02-06T21:40:36.658 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.655+0000 7ffa1ed9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa20142680 0x7ffa20142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:36.658 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.655+0000 7ffa1e59c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffa200b54b0 0x7ffa200a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:36.658 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1ed9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa20142680 0x7ffa20142ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.135:37164/0 (socket says 172.21.15.135:37164) 2024-02-06T21:40:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1ed9d640 1 -- 172.21.15.135:0/1523146886 learned_addr learned my addr 172.21.15.135:0/1523146886 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-02-06T21:40:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1ed9d640 10 monclient(hunting): get_auth_request con 0x7ffa20142680 auth_method 0 2024-02-06T21:40:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1ed9d640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1e59c640 1 --2- 172.21.15.135:0/1523146886 >> v2:172.21.15.120:3300/0 conn(0x7ffa200b54b0 0x7ffa200a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.135:37142/0 (socket says 172.21.15.135:37142) 2024-02-06T21:40:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1e59c640 10 monclient(hunting): get_auth_request con 0x7ffa200b54b0 auth_method 0 2024-02-06T21:40:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1e59c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:36.660 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.656+0000 7ffa1ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:36.660 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.658+0000 7ffa1e59c640 10 monclient(hunting): handle_auth_done global_id 4146 payload 274 2024-02-06T21:40:36.660 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.658+0000 7ffa1e59c640 1 -- 172.21.15.135:0/1523146886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa20142680 msgr2=0x7ffa20142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:36.660 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.658+0000 7ffa1e59c640 1 --2- 172.21.15.135:0/1523146886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa20142680 0x7ffa20142ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.661 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.658+0000 7ffa1e59c640 1 -- 172.21.15.135:0/1523146886 >> v1:172.21.15.120:6789/0 conn(0x7ffa2013f830 legacy=0x7ffa2013fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:36.661 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.658+0000 7ffa1e59c640 10 monclient: _finish_hunting 0 2024-02-06T21:40:36.661 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.658+0000 7ffa1e59c640 1 monclient: found mon.noname-a 2024-02-06T21:40:36.661 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.658+0000 7ffa1e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:36.661 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.658+0000 7ffa1e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:40:36.661 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.658+0000 7ffa1e59c640 1 -- 172.21.15.135:0/1523146886 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa200b4800 con 0x7ffa200b54b0 2024-02-06T21:40:36.661 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.661+0000 7ffa1e59c640 1 --2- 172.21.15.135:0/1523146886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa200b54b0 0x7ffa200a9860 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ffa100097d0 tx=0x7ffa1002b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=275bcc785a2b2e5b server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:36.667 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa1d59a640 1 -- 172.21.15.135:0/1523146886 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa10031770 con 0x7ffa200b54b0 2024-02-06T21:40:36.667 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa1d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:36.667 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa1d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:40:36.667 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa1d59a640 10 monclient: dump: 2024-02-06T21:40:36.667 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:36.667 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:36.667 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.667 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.667 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:36.667 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:36.667 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:36.668 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:36.668 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:36.668 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:36.668 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa1d59a640 10 monclient: _finish_auth 0 2024-02-06T21:40:36.668 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa1d59a640 10 monclient: _check_auth_tickets 2024-02-06T21:40:36.668 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa1d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:36.668 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa1d59a640 1 -- 172.21.15.135:0/1523146886 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa100318d0 con 0x7ffa200b54b0 2024-02-06T21:40:36.668 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa1d59a640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:36.668 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa1d59a640 1 -- 172.21.15.135:0/1523146886 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa10031c60 con 0x7ffa200b54b0 2024-02-06T21:40:36.668 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa1d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:36.668 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa1d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:40:36.668 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa1d59a640 10 monclient: dump: 2024-02-06T21:40:36.668 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:36.668 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:36.668 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.669 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.669 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:36.669 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:36.669 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:36.669 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:36.669 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:36.669 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:36.669 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa248e0640 5 monclient: authenticate success, global_id 4146 2024-02-06T21:40:36.669 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa248e0640 10 monclient: get_monmap_and_config success 2024-02-06T21:40:36.669 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.665+0000 7ffa248e0640 4 set_mon_vals no callback set 2024-02-06T21:40:36.669 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.666+0000 7ffa248e0640 10 monclient: shutdown 2024-02-06T21:40:36.669 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.666+0000 7ffa248e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:36.669 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.666+0000 7ffa248e0640 1 -- 172.21.15.135:0/1523146886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa200b54b0 msgr2=0x7ffa200a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:36.669 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.666+0000 7ffa248e0640 1 --2- 172.21.15.135:0/1523146886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa200b54b0 0x7ffa200a9860 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ffa100097d0 tx=0x7ffa1002b810 comp rx=0 tx=0).stop 2024-02-06T21:40:36.669 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.666+0000 7ffa248e0640 1 -- 172.21.15.135:0/1523146886 shutdown_connections 2024-02-06T21:40:36.670 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.666+0000 7ffa248e0640 1 --2- 172.21.15.135:0/1523146886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa20142680 0x7ffa20142ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.670 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.666+0000 7ffa248e0640 1 --2- 172.21.15.135:0/1523146886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa200b54b0 0x7ffa200a9860 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.670 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.666+0000 7ffa248e0640 1 -- 172.21.15.135:0/1523146886 >> 172.21.15.135:0/1523146886 conn(0x7ffa200b8660 msgr2=0x7ffa200b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:36.670 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.666+0000 7ffa248e0640 1 -- 172.21.15.135:0/1523146886 shutdown_connections 2024-02-06T21:40:36.670 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.666+0000 7ffa248e0640 1 -- 172.21.15.135:0/1523146886 wait complete. 2024-02-06T21:40:36.673 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.667+0000 7ffa248e0640 10 monclient: build_initial_monmap 2024-02-06T21:40:36.673 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.667+0000 7ffa248e0640 10 monclient: monmap: 2024-02-06T21:40:36.673 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:36.673 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:40:36.673 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2024-02-06T21:40:36.674 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2024-02-06T21:40:36.674 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:36.674 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:36.674 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:40:36.674 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:40:36.674 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:40:36.674 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:36.674 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.667+0000 7ffa248e0640 1 librados: starting msgr at 2024-02-06T21:40:36.674 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.667+0000 7ffa248e0640 1 librados: starting objecter 2024-02-06T21:40:36.674 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.667+0000 7ffa248e0640 1 Processor -- start 2024-02-06T21:40:36.674 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.667+0000 7ffa248e0640 1 -- start start 2024-02-06T21:40:36.674 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.667+0000 7ffa248e0640 1 librados: setting wanted keys 2024-02-06T21:40:36.674 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.667+0000 7ffa248e0640 1 librados: calling monclient init 2024-02-06T21:40:36.674 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.667+0000 7ffa248e0640 10 monclient: init 2024-02-06T21:40:36.674 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:36.675 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T21:40:36.675 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa200b54b0 0x7ffa20135a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:36.675 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 10 monclient(hunting): picked mon.noname-c con 0x7ffa200b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:36.675 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa2013f830 0x7ffa201dba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:36.675 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 10 monclient(hunting): picked mon.noname-b con 0x7ffa2013f830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:40:36.675 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa20142680 0x7ffa201336f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:36.675 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 10 monclient(hunting): picked mon.noname-a con 0x7ffa20142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:40:36.675 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:36.675 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa201353e0 con 0x7ffa20142680 2024-02-06T21:40:36.675 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:36.675 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa201381a0 con 0x7ffa2013f830 2024-02-06T21:40:36.675 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:36.675 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa201db560 con 0x7ffa200b54b0 2024-02-06T21:40:36.675 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:36.675 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa248e0640 10 monclient(hunting): authenticate will time out at 1277.990085s 2024-02-06T21:40:36.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa1dd9b640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa2013f830 0x7ffa201dba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:36.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa1ed9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa20142680 0x7ffa201336f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:36.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.668+0000 7ffa1e59c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa200b54b0 0x7ffa20135a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:36.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1dd9b640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa2013f830 0x7ffa201dba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:47502/0 (socket says 172.21.15.135:47502) 2024-02-06T21:40:36.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1dd9b640 1 -- 172.21.15.135:0/1650051144 learned_addr learned my addr 172.21.15.135:0/1650051144 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-02-06T21:40:36.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1ed9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa20142680 0x7ffa201336f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.135:37144/0 (socket says 172.21.15.135:37144) 2024-02-06T21:40:36.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1dd9b640 10 monclient(hunting): get_auth_request con 0x7ffa2013f830 auth_method 0 2024-02-06T21:40:36.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:36.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1dd9b640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:36.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:36.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1ed9d640 10 monclient(hunting): get_auth_request con 0x7ffa20142680 auth_method 0 2024-02-06T21:40:36.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:36.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1ed9d640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:36.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1e59c640 10 monclient(hunting): get_auth_request con 0x7ffa200b54b0 auth_method 0 2024-02-06T21:40:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1e59c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.669+0000 7ffa1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.672+0000 7ffa1ed9d640 10 monclient(hunting): handle_auth_done global_id 4149 payload 824 2024-02-06T21:40:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.672+0000 7ffa1ed9d640 1 -- 172.21.15.135:0/1650051144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa200b54b0 msgr2=0x7ffa20135a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:36.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.672+0000 7ffa1ed9d640 1 --2- 172.21.15.135:0/1650051144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa200b54b0 0x7ffa20135a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.672+0000 7ffa1ed9d640 1 -- 172.21.15.135:0/1650051144 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa2013f830 msgr2=0x7ffa201dba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:36.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.672+0000 7ffa1ed9d640 1 --2- 172.21.15.135:0/1650051144 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa2013f830 0x7ffa201dba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.672+0000 7ffa1ed9d640 10 monclient: _finish_hunting 0 2024-02-06T21:40:36.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.672+0000 7ffa1ed9d640 1 monclient: found mon.noname-a 2024-02-06T21:40:36.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.672+0000 7ffa1ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:36.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.672+0000 7ffa1ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:40:36.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.672+0000 7ffa1ed9d640 1 -- 172.21.15.135:0/1650051144 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa10009400 con 0x7ffa20142680 2024-02-06T21:40:36.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.672+0000 7ffa1dd9b640 1 --2- 172.21.15.135:0/1650051144 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa2013f830 0x7ffa201dba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-06T21:40:36.680 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.674+0000 7ffa1ed9d640 1 --2- 172.21.15.135:0/1650051144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa20142680 0x7ffa201336f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800eba0 tx=0x7ffa1800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:36.680 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.674+0000 7ffa0effd640 10 client.?.objecter ms_handle_connect 0x7ffa20142680 2024-02-06T21:40:36.680 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.674+0000 7ffa0effd640 10 client.?.objecter resend_mon_ops 2024-02-06T21:40:36.680 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa0effd640 1 -- 172.21.15.135:0/1650051144 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1800cb20 con 0x7ffa20142680 2024-02-06T21:40:36.680 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:36.680 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa0effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:40:36.680 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa0effd640 10 monclient: dump: 2024-02-06T21:40:36.680 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:36.680 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:36.681 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.681 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.681 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:36.681 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:36.681 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:36.681 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:36.681 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:36.681 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:36.681 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa0effd640 10 monclient: _finish_auth 0 2024-02-06T21:40:36.681 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa0effd640 10 monclient: _check_auth_tickets 2024-02-06T21:40:36.681 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:36.681 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa248e0640 5 monclient: authenticate success, global_id 4149 2024-02-06T21:40:36.681 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa0effd640 1 -- 172.21.15.135:0/1650051144 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1800f040 con 0x7ffa20142680 2024-02-06T21:40:36.681 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa248e0640 10 monclient: _renew_subs 2024-02-06T21:40:36.681 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa248e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:36.682 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa248e0640 1 -- 172.21.15.135:0/1650051144 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa200b1b10 con 0x7ffa20142680 2024-02-06T21:40:36.682 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa0effd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:36.682 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa0effd640 1 -- 172.21.15.135:0/1650051144 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1800cea0 con 0x7ffa20142680 2024-02-06T21:40:36.682 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:36.682 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:40:36.682 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa0effd640 10 monclient: dump: 2024-02-06T21:40:36.682 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:36.682 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:36.682 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.682 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:36.682 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:36.682 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:36.682 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:36.682 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:36.682 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:36.683 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:36.683 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa1cd99640 4 set_mon_vals no callback set 2024-02-06T21:40:36.683 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa248e0640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:40:36.683 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa248e0640 10 monclient: _renew_subs 2024-02-06T21:40:36.683 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa248e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:36.683 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa248e0640 1 -- 172.21.15.135:0/1650051144 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa201391d0 con 0x7ffa20142680 2024-02-06T21:40:36.683 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.678+0000 7ffa248e0640 1 librados: init done 2024-02-06T21:40:36.683 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.679+0000 7ffa248e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:36.683 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.679+0000 7ffa248e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:36.683 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.679+0000 7ffa248e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:36.683 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.679+0000 7ffa248e0640 1 -- 172.21.15.135:0/1650051144 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa20139450 con 0x7ffa20142680 2024-02-06T21:40:36.843 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.843+0000 7ffa0effd640 1 -- 172.21.15.135:0/1650051144 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ffa18003990 con 0x7ffa20142680 2024-02-06T21:40:36.848 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.847+0000 7ffa0effd640 1 -- 172.21.15.135:0/1650051144 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7ffa18014a30 con 0x7ffa20142680 2024-02-06T21:40:36.848 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.847+0000 7ffa0effd640 10 client.4149.objecter ms_dispatch 0x7ffa20000b90 osd_map(8..8 src has 1..8) v4 2024-02-06T21:40:36.848 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.847+0000 7ffa0effd640 3 client.4149.objecter handle_osd_map got epochs [8,8] > 0 2024-02-06T21:40:36.848 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.847+0000 7ffa0effd640 3 client.4149.objecter handle_osd_map decoding full epoch 8 2024-02-06T21:40:36.848 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.847+0000 7ffa0effd640 20 client.4149.objecter dump_active .. 0 homeless 2024-02-06T21:40:36.848 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.847+0000 7ffa0effd640 1 -- 172.21.15.135:0/1650051144 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ffa18003e20 con 0x7ffa20142680 2024-02-06T21:40:36.849 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.847+0000 7ffa0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:36.849 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.847+0000 7ffa0effd640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:40:36.920 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.919+0000 7ffa248e0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1d6b9f96-1f9d-44f2-83b0-93f5c342fafb", "id": 6}] 2024-02-06T21:40:36.921 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.919+0000 7ffa248e0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1d6b9f96-1f9d-44f2-83b0-93f5c342fafb", "id": 6}] 2024-02-06T21:40:36.921 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.919+0000 7ffa248e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:36.921 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.919+0000 7ffa248e0640 1 -- 172.21.15.135:0/1650051144 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1d6b9f96-1f9d-44f2-83b0-93f5c342fafb", "id": 6} v 0) v1 -- 0x7ffa200a9c00 con 0x7ffa20142680 2024-02-06T21:40:36.923 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:36.922+0000 7f05e902c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-06T21:40:36.971 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.970+0000 7ffa0effd640 1 -- 172.21.15.135:0/1650051144 <== mon.0 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1d6b9f96-1f9d-44f2-83b0-93f5c342fafb", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7ffa200a9c00 con 0x7ffa20142680 2024-02-06T21:40:36.971 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.970+0000 7ffa0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1d6b9f96-1f9d-44f2-83b0-93f5c342fafb", "id": 6}] 2024-02-06T21:40:36.971 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.970+0000 7ffa0effd640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:40:36.971 INFO:teuthology.orchestra.run.smithi135.stdout:6 2024-02-06T21:40:36.973 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 10 librados: watch_flush enter 2024-02-06T21:40:36.973 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 10 librados: watch_flush exit 2024-02-06T21:40:36.973 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 20 client.4149.objecter shutdown clearing up homeless session... 2024-02-06T21:40:36.973 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 10 client.4149.objecter successfully canceled tick 2024-02-06T21:40:36.973 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 10 monclient: shutdown 2024-02-06T21:40:36.973 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:36.973 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 1 -- 172.21.15.135:0/1650051144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa20142680 msgr2=0x7ffa201336f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:36.973 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 1 --2- 172.21.15.135:0/1650051144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa20142680 0x7ffa201336f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800eba0 tx=0x7ffa1800c490 comp rx=0 tx=0).stop 2024-02-06T21:40:36.973 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 1 -- 172.21.15.135:0/1650051144 shutdown_connections 2024-02-06T21:40:36.973 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 1 --2- 172.21.15.135:0/1650051144 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa2013f830 0x7ffa201dba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.974 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 1 --2- 172.21.15.135:0/1650051144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa20142680 0x7ffa201336f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.974 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 1 --2- 172.21.15.135:0/1650051144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa200b54b0 0x7ffa20135a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:36.974 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 1 -- 172.21.15.135:0/1650051144 >> 172.21.15.135:0/1650051144 conn(0x7ffa200b8660 msgr2=0x7ffa200744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:36.974 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 1 -- 172.21.15.135:0/1650051144 shutdown_connections 2024-02-06T21:40:36.974 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 1 -- 172.21.15.135:0/1650051144 wait complete. 2024-02-06T21:40:36.974 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:36.971+0000 7ffa248e0640 1 librados: shutdown 2024-02-06T21:40:36.978 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:36.977+0000 7f05e902c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-06T21:40:36.983 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new ecae00f5-8e29-47dd-90d5-e51503bc9dbd 7 2024-02-06T21:40:36.998 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:36.997+0000 7f4683038040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-06T21:40:37.082 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.081+0000 7ff6f4754640 10 monclient: get_monmap_and_config 2024-02-06T21:40:37.082 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.081+0000 7ff6f4754640 10 monclient: build_initial_monmap 2024-02-06T21:40:37.082 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.081+0000 7ff6f4754640 10 monclient: monmap: 2024-02-06T21:40:37.082 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:37.082 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:37.082 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:40:37.081781+0000 2024-02-06T21:40:37.083 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:40:37.081781+0000 2024-02-06T21:40:37.083 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:37.083 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:37.083 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:40:37.083 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:40:37.083 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:40:37.083 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:40:37.083 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:40:37.083 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:37.083 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.081+0000 7ff6f4754640 1 Processor -- start 2024-02-06T21:40:37.083 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.081+0000 7ff6f4754640 1 -- start start 2024-02-06T21:40:37.083 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.081+0000 7ff6f4754640 10 monclient: init 2024-02-06T21:40:37.086 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:37.086 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-02-06T21:40:37.086 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff6ec0b54b0 0x7ff6ec0a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:37.086 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 10 monclient(hunting): picked mon.noname-c con 0x7ff6ec0b54b0 addr v2:172.21.15.135:3300/0 2024-02-06T21:40:37.086 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6ec13f830 0x7ff6ec13fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:37.086 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 10 monclient(hunting): picked mon.noname-b con 0x7ff6ec13f830 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:37.086 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff6ec142680 0x7ff6ec142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:37.086 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 10 monclient(hunting): picked mon.noname-a con 0x7ff6ec142680 addr v2:172.21.15.120:3300/0 2024-02-06T21:40:37.086 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:37.086 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6ec0ab520 con 0x7ff6ec142680 2024-02-06T21:40:37.086 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:37.086 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6ec0b5890 con 0x7ff6ec0b54b0 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec0aa450 con 0x7ff6ec13f830 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.082+0000 7ff6f4754640 10 monclient(hunting): authenticate will time out at 1278.404553s 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f24f0640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff6ec0b54b0 0x7ff6ec0a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f1cef640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6ec13f830 0x7ff6ec13fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f2cf1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff6ec142680 0x7ff6ec142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f24f0640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff6ec0b54b0 0x7ff6ec0a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:47514/0 (socket says 172.21.15.135:47514) 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f24f0640 1 -- 172.21.15.135:0/4131156181 learned_addr learned my addr 172.21.15.135:0/4131156181 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f24f0640 10 monclient(hunting): get_auth_request con 0x7ff6ec0b54b0 auth_method 0 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f24f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f24f0640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f24f0640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f2cf1640 10 monclient(hunting): get_auth_request con 0x7ff6ec142680 auth_method 0 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f2cf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:37.087 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f2cf1640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:37.088 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f2cf1640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:37.088 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f1cef640 10 monclient(hunting): get_auth_request con 0x7ff6ec13f830 auth_method 0 2024-02-06T21:40:37.088 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f1cef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:37.088 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f1cef640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:37.088 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.083+0000 7ff6f1cef640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:37.088 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.084+0000 7ff6f2cf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:37.088 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.084+0000 7ff6f2cf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:37.088 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.084+0000 7ff6f2cf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:37.088 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.084+0000 7ff6f24f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:37.088 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.084+0000 7ff6f24f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:37.088 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.084+0000 7ff6f24f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:37.088 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.084+0000 7ff6f1cef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:37.088 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.084+0000 7ff6f1cef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:37.088 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.084+0000 7ff6f1cef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:37.089 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.085+0000 7ff6f2cf1640 10 monclient(hunting): handle_auth_done global_id 4152 payload 274 2024-02-06T21:40:37.090 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.085+0000 7ff6f2cf1640 1 -- 172.21.15.135:0/4131156181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6ec13f830 msgr2=0x7ff6ec13fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:37.090 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.085+0000 7ff6f2cf1640 1 --2- 172.21.15.135:0/4131156181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6ec13f830 0x7ff6ec13fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:37.090 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.085+0000 7ff6f2cf1640 1 -- 172.21.15.135:0/4131156181 >> v2:172.21.15.135:3300/0 conn(0x7ff6ec0b54b0 msgr2=0x7ff6ec0a9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:37.090 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.085+0000 7ff6f2cf1640 1 --2- 172.21.15.135:0/4131156181 >> v2:172.21.15.135:3300/0 conn(0x7ff6ec0b54b0 0x7ff6ec0a9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:37.090 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.085+0000 7ff6f2cf1640 10 monclient: _finish_hunting 0 2024-02-06T21:40:37.090 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.085+0000 7ff6f2cf1640 1 monclient: found mon.noname-a 2024-02-06T21:40:37.090 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.085+0000 7ff6f2cf1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:37.090 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.085+0000 7ff6f2cf1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:40:37.090 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.085+0000 7ff6f2cf1640 1 -- 172.21.15.135:0/4131156181 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec0ab720 con 0x7ff6ec142680 2024-02-06T21:40:37.090 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.087+0000 7ff6f2cf1640 1 --2- 172.21.15.135:0/4131156181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6ec142680 0x7ff6ec142af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec06c7b0 tx=0x7ff6e402be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d69c0c596370caff server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:37.093 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f14ee640 1 -- 172.21.15.135:0/4131156181 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e402cbb0 con 0x7ff6ec142680 2024-02-06T21:40:37.093 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f14ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:37.093 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f14ee640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:40:37.093 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f14ee640 10 monclient: dump: 2024-02-06T21:40:37.093 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:37.093 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:37.093 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.093 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.093 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:37.093 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:37.093 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:37.094 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:37.094 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:37.094 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:37.094 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f14ee640 10 monclient: _finish_auth 0 2024-02-06T21:40:37.094 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f14ee640 10 monclient: _check_auth_tickets 2024-02-06T21:40:37.094 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f14ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:37.094 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f14ee640 1 -- 172.21.15.135:0/4131156181 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e402cd10 con 0x7ff6ec142680 2024-02-06T21:40:37.094 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f4754640 5 monclient: authenticate success, global_id 4152 2024-02-06T21:40:37.094 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f4754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:40:37.094 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f14ee640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:37.094 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f4754640 10 monclient: get_monmap_and_config success 2024-02-06T21:40:37.094 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f4754640 4 set_mon_vals no callback set 2024-02-06T21:40:37.094 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f14ee640 1 -- 172.21.15.135:0/4131156181 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4034660 con 0x7ff6ec142680 2024-02-06T21:40:37.094 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f14ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:37.094 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f14ee640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:40:37.095 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f14ee640 10 monclient: dump: 2024-02-06T21:40:37.095 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:37.095 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:37.095 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.095 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.095 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:37.095 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:37.095 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:37.095 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:37.095 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:37.095 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:37.100 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f4754640 10 monclient: shutdown 2024-02-06T21:40:37.100 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f4754640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:37.100 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f4754640 1 -- 172.21.15.135:0/4131156181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6ec142680 msgr2=0x7ff6ec142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:37.100 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f4754640 1 --2- 172.21.15.135:0/4131156181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6ec142680 0x7ff6ec142af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec06c7b0 tx=0x7ff6e402be00 comp rx=0 tx=0).stop 2024-02-06T21:40:37.100 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f4754640 1 -- 172.21.15.135:0/4131156181 shutdown_connections 2024-02-06T21:40:37.100 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f4754640 1 --2- 172.21.15.135:0/4131156181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6ec142680 0x7ff6ec142af0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:37.100 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f4754640 1 --2- 172.21.15.135:0/4131156181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6ec13f830 0x7ff6ec13fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:37.100 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f4754640 1 --2- 172.21.15.135:0/4131156181 >> v2:172.21.15.135:3300/0 conn(0x7ff6ec0b54b0 0x7ff6ec0a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:37.100 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f4754640 1 -- 172.21.15.135:0/4131156181 >> 172.21.15.135:0/4131156181 conn(0x7ff6ec0b8660 msgr2=0x7ff6ec0b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:37.100 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.092+0000 7ff6f4754640 1 -- 172.21.15.135:0/4131156181 shutdown_connections 2024-02-06T21:40:37.100 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.093+0000 7ff6f4754640 1 -- 172.21.15.135:0/4131156181 wait complete. 2024-02-06T21:40:37.100 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.093+0000 7ff6f4754640 10 monclient: build_initial_monmap 2024-02-06T21:40:37.101 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.093+0000 7ff6f4754640 10 monclient: monmap: 2024-02-06T21:40:37.101 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-02-06T21:40:37.101 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:40:37.101 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2024-02-06T21:40:37.101 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2024-02-06T21:40:37.101 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:37.101 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:37.101 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:40:37.101 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:40:37.101 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:40:37.101 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:37.101 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.093+0000 7ff6f4754640 1 librados: starting msgr at 2024-02-06T21:40:37.101 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.093+0000 7ff6f4754640 1 librados: starting objecter 2024-02-06T21:40:37.101 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.093+0000 7ff6f4754640 1 Processor -- start 2024-02-06T21:40:37.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 1 -- start start 2024-02-06T21:40:37.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 1 librados: setting wanted keys 2024-02-06T21:40:37.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 1 librados: calling monclient init 2024-02-06T21:40:37.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 10 monclient: init 2024-02-06T21:40:37.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:37.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 10 monclient: _add_conns ranks=[0,1,2] 2024-02-06T21:40:37.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6ec0b54b0 0x7ff6ec0b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:37.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 10 monclient(hunting): picked mon.noname-a con 0x7ff6ec0b54b0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:40:37.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6ec13f830 0x7ff6ec0af790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:37.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 10 monclient(hunting): picked mon.noname-b con 0x7ff6ec13f830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:40:37.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6ec142680 0x7ff6ec0ae9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:37.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 10 monclient(hunting): picked mon.noname-c con 0x7ff6ec142680 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:37.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:37.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec0b0f10 con 0x7ff6ec0b54b0 2024-02-06T21:40:37.102 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:37.103 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec0af300 con 0x7ff6ec13f830 2024-02-06T21:40:37.103 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:37.103 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec138c10 con 0x7ff6ec142680 2024-02-06T21:40:37.103 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:37.103 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.094+0000 7ff6f4754640 10 monclient(hunting): authenticate will time out at 1278.416605s 2024-02-06T21:40:37.103 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f1cef640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6ec13f830 0x7ff6ec0af790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:37.103 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f24f0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6ec0b54b0 0x7ff6ec0b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:37.103 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f2cf1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6ec142680 0x7ff6ec0ae9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:37.103 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f24f0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6ec0b54b0 0x7ff6ec0b1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.135:37164/0 (socket says 172.21.15.135:37164) 2024-02-06T21:40:37.103 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f24f0640 1 -- 172.21.15.135:0/1585934659 learned_addr learned my addr 172.21.15.135:0/1585934659 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-02-06T21:40:37.103 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f24f0640 10 monclient(hunting): get_auth_request con 0x7ff6ec0b54b0 auth_method 0 2024-02-06T21:40:37.103 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f24f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:37.103 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f24f0640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:37.104 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f24f0640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:37.104 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f1cef640 10 monclient(hunting): get_auth_request con 0x7ff6ec13f830 auth_method 0 2024-02-06T21:40:37.104 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f1cef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:37.104 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f1cef640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:37.104 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f1cef640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:37.104 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f2cf1640 10 monclient(hunting): get_auth_request con 0x7ff6ec142680 auth_method 0 2024-02-06T21:40:37.104 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f2cf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:37.104 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f2cf1640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:37.104 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f2cf1640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:37.104 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f24f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:37.104 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f24f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:37.104 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.095+0000 7ff6f24f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:37.104 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.096+0000 7ff6f1cef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:37.104 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.096+0000 7ff6f1cef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:37.105 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.096+0000 7ff6f1cef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:37.105 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.096+0000 7ff6f2cf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:37.105 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.096+0000 7ff6f2cf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:37.105 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.096+0000 7ff6f2cf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:37.105 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.098+0000 7ff6f24f0640 10 monclient(hunting): handle_auth_done global_id 4155 payload 824 2024-02-06T21:40:37.105 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.098+0000 7ff6f24f0640 1 -- 172.21.15.135:0/1585934659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6ec142680 msgr2=0x7ff6ec0ae9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:37.105 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.098+0000 7ff6f24f0640 1 --2- 172.21.15.135:0/1585934659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6ec142680 0x7ff6ec0ae9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:37.105 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.098+0000 7ff6f24f0640 1 -- 172.21.15.135:0/1585934659 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6ec13f830 msgr2=0x7ff6ec0af790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:37.105 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.098+0000 7ff6f24f0640 1 --2- 172.21.15.135:0/1585934659 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6ec13f830 0x7ff6ec0af790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:37.105 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.098+0000 7ff6f24f0640 10 monclient: _finish_hunting 0 2024-02-06T21:40:37.105 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.098+0000 7ff6f24f0640 1 monclient: found mon.noname-a 2024-02-06T21:40:37.105 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.098+0000 7ff6f24f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:37.105 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.098+0000 7ff6f24f0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:40:37.105 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.098+0000 7ff6f24f0640 1 -- 172.21.15.135:0/1585934659 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e4009a70 con 0x7ff6ec0b54b0 2024-02-06T21:40:37.106 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.101+0000 7ff6f24f0640 1 --2- 172.21.15.135:0/1585934659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6ec0b54b0 0x7ff6ec0b1250 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00e840 tx=0x7ff6dc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:37.106 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.101+0000 7ff6e2ffd640 10 client.?.objecter ms_handle_connect 0x7ff6ec0b54b0 2024-02-06T21:40:37.106 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.101+0000 7ff6e2ffd640 10 client.?.objecter resend_mon_ops 2024-02-06T21:40:37.107 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:37.105+0000 7f05e902c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-06T21:40:37.108 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6e2ffd640 1 -- 172.21.15.135:0/1585934659 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc00c8e0 con 0x7ff6ec0b54b0 2024-02-06T21:40:37.108 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:37.108 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:40:37.108 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6e2ffd640 10 monclient: dump: 2024-02-06T21:40:37.108 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:37.108 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:37.108 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.108 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.108 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:37.108 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:37.108 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:37.108 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:37.108 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:37.109 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:37.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6e2ffd640 10 monclient: _finish_auth 0 2024-02-06T21:40:37.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6e2ffd640 10 monclient: _check_auth_tickets 2024-02-06T21:40:37.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:37.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6e2ffd640 1 -- 172.21.15.135:0/1585934659 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc00ca40 con 0x7ff6ec0b54b0 2024-02-06T21:40:37.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:37.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6e2ffd640 1 -- 172.21.15.135:0/1585934659 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc00cdf0 con 0x7ff6ec0b54b0 2024-02-06T21:40:37.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6f4754640 5 monclient: authenticate success, global_id 4155 2024-02-06T21:40:37.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6f4754640 10 monclient: _renew_subs 2024-02-06T21:40:37.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6f4754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:37.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6f4754640 1 -- 172.21.15.135:0/1585934659 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ec0b17f0 con 0x7ff6ec0b54b0 2024-02-06T21:40:37.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:37.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6e3fff640 4 set_mon_vals no callback set 2024-02-06T21:40:37.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:40:37.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6e2ffd640 10 monclient: dump: 2024-02-06T21:40:37.110 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-02-06T21:40:37.110 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:37.110 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.110 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.110 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:37.110 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-02-06T21:40:37.110 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:37.110 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:37.110 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:37.110 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T21:40:37.110 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6f4754640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:40:37.110 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6f4754640 10 monclient: _renew_subs 2024-02-06T21:40:37.110 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6f4754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:37.110 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6f4754640 1 -- 172.21.15.135:0/1585934659 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ec0aef10 con 0x7ff6ec0b54b0 2024-02-06T21:40:37.110 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.105+0000 7ff6f4754640 1 librados: init done 2024-02-06T21:40:37.111 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.106+0000 7ff6f4754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:37.111 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.106+0000 7ff6f4754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:37.111 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.106+0000 7ff6f4754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:37.111 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.106+0000 7ff6f4754640 1 -- 172.21.15.135:0/1585934659 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6ec04ef90 con 0x7ff6ec0b54b0 2024-02-06T21:40:37.247 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.246+0000 7ff6e2ffd640 1 -- 172.21.15.135:0/1585934659 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff6dc005990 con 0x7ff6ec0b54b0 2024-02-06T21:40:37.251 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.251+0000 7ff6e2ffd640 1 -- 172.21.15.135:0/1585934659 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7ff6dc0067d0 con 0x7ff6ec0b54b0 2024-02-06T21:40:37.252 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.251+0000 7ff6e2ffd640 10 client.4155.objecter ms_dispatch 0x7ff6ec000b90 osd_map(9..9 src has 1..9) v4 2024-02-06T21:40:37.252 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.251+0000 7ff6e2ffd640 3 client.4155.objecter handle_osd_map got epochs [9,9] > 0 2024-02-06T21:40:37.252 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.251+0000 7ff6e2ffd640 3 client.4155.objecter handle_osd_map decoding full epoch 9 2024-02-06T21:40:37.252 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.251+0000 7ff6e2ffd640 20 client.4155.objecter dump_active .. 0 homeless 2024-02-06T21:40:37.252 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.251+0000 7ff6e2ffd640 1 -- 172.21.15.135:0/1585934659 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff6dc021060 con 0x7ff6ec0b54b0 2024-02-06T21:40:37.252 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.251+0000 7ff6e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:37.252 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.251+0000 7ff6e2ffd640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:40:37.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.320+0000 7ff6f4754640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ecae00f5-8e29-47dd-90d5-e51503bc9dbd", "id": 7}] 2024-02-06T21:40:37.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.320+0000 7ff6f4754640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ecae00f5-8e29-47dd-90d5-e51503bc9dbd", "id": 7}] 2024-02-06T21:40:37.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.320+0000 7ff6f4754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:37.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.320+0000 7ff6f4754640 1 -- 172.21.15.135:0/1585934659 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ecae00f5-8e29-47dd-90d5-e51503bc9dbd", "id": 7} v 0) v1 -- 0x7ff6ec13a340 con 0x7ff6ec0b54b0 2024-02-06T21:40:37.321 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:37.319+0000 7f4683038040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-06T21:40:37.378 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:37.376+0000 7f4683038040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-06T21:40:37.439 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.438+0000 7ff6e2ffd640 1 -- 172.21.15.135:0/1585934659 <== mon.0 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ecae00f5-8e29-47dd-90d5-e51503bc9dbd", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff6dc017070 con 0x7ff6ec0b54b0 2024-02-06T21:40:37.439 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.438+0000 7ff6e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ecae00f5-8e29-47dd-90d5-e51503bc9dbd", "id": 7}] 2024-02-06T21:40:37.439 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.438+0000 7ff6e2ffd640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:40:37.439 INFO:teuthology.orchestra.run.smithi135.stdout:7 2024-02-06T21:40:37.441 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 10 librados: watch_flush enter 2024-02-06T21:40:37.441 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 10 librados: watch_flush exit 2024-02-06T21:40:37.441 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 20 client.4155.objecter shutdown clearing up homeless session... 2024-02-06T21:40:37.441 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 10 client.4155.objecter successfully canceled tick 2024-02-06T21:40:37.441 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 10 monclient: shutdown 2024-02-06T21:40:37.441 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:37.441 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 1 -- 172.21.15.135:0/1585934659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6ec0b54b0 msgr2=0x7ff6ec0b1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:37.441 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 1 --2- 172.21.15.135:0/1585934659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6ec0b54b0 0x7ff6ec0b1250 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00e840 tx=0x7ff6dc00ed00 comp rx=0 tx=0).stop 2024-02-06T21:40:37.441 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 1 -- 172.21.15.135:0/1585934659 shutdown_connections 2024-02-06T21:40:37.441 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 1 --2- 172.21.15.135:0/1585934659 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6ec13f830 0x7ff6ec0af790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:37.441 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 1 --2- 172.21.15.135:0/1585934659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6ec142680 0x7ff6ec0ae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:37.441 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 1 --2- 172.21.15.135:0/1585934659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6ec0b54b0 0x7ff6ec0b1250 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:37.442 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 1 -- 172.21.15.135:0/1585934659 >> 172.21.15.135:0/1585934659 conn(0x7ff6ec0b8660 msgr2=0x7ff6ec0a9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:37.442 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 1 -- 172.21.15.135:0/1585934659 shutdown_connections 2024-02-06T21:40:37.442 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 1 -- 172.21.15.135:0/1585934659 wait complete. 2024-02-06T21:40:37.442 INFO:teuthology.orchestra.run.smithi135.stderr:2024-02-06T21:40:37.439+0000 7ff6f4754640 1 librados: shutdown 2024-02-06T21:40:37.450 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-02-06T21:40:37.450 INFO:tasks.ceph.osd.0:Restarting daemon 2024-02-06T21:40:37.451 DEBUG:teuthology.orchestra.run.smithi120:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-02-06T21:40:37.453 INFO:tasks.ceph.osd.0:Started 2024-02-06T21:40:37.453 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-02-06T21:40:37.453 INFO:tasks.ceph.osd.1:Restarting daemon 2024-02-06T21:40:37.453 DEBUG:teuthology.orchestra.run.smithi120:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-02-06T21:40:37.455 INFO:tasks.ceph.osd.1:Started 2024-02-06T21:40:37.456 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-02-06T21:40:37.456 INFO:tasks.ceph.osd.2:Restarting daemon 2024-02-06T21:40:37.456 DEBUG:teuthology.orchestra.run.smithi120:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-02-06T21:40:37.458 INFO:tasks.ceph.osd.2:Started 2024-02-06T21:40:37.458 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-02-06T21:40:37.458 INFO:tasks.ceph.osd.3:Restarting daemon 2024-02-06T21:40:37.458 DEBUG:teuthology.orchestra.run.smithi120:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-02-06T21:40:37.464 INFO:tasks.ceph.osd.3:Started 2024-02-06T21:40:37.464 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-02-06T21:40:37.464 INFO:tasks.ceph.osd.4:Restarting daemon 2024-02-06T21:40:37.464 DEBUG:teuthology.orchestra.run.smithi135:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-02-06T21:40:37.467 INFO:tasks.ceph.osd.4:Started 2024-02-06T21:40:37.467 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-02-06T21:40:37.467 INFO:tasks.ceph.osd.5:Restarting daemon 2024-02-06T21:40:37.467 DEBUG:teuthology.orchestra.run.smithi135:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-02-06T21:40:37.469 INFO:tasks.ceph.osd.5:Started 2024-02-06T21:40:37.469 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-02-06T21:40:37.469 INFO:tasks.ceph.osd.6:Restarting daemon 2024-02-06T21:40:37.469 DEBUG:teuthology.orchestra.run.smithi135:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-02-06T21:40:37.472 INFO:tasks.ceph.osd.6:Started 2024-02-06T21:40:37.472 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-02-06T21:40:37.472 INFO:tasks.ceph.osd.7:Restarting daemon 2024-02-06T21:40:37.472 DEBUG:teuthology.orchestra.run.smithi135:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-02-06T21:40:37.474 INFO:tasks.ceph.osd.7:Started 2024-02-06T21:40:37.474 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-06T21:40:37.544 INFO:tasks.ceph.mgr.y.smithi120.stderr:2024-02-06T21:40:37.543+0000 7f4683038040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-06T21:40:37.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.643+0000 7f4036a74640 10 monclient: get_monmap_and_config 2024-02-06T21:40:37.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.643+0000 7f4036a74640 10 monclient: build_initial_monmap 2024-02-06T21:40:37.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.643+0000 7f4036a74640 10 monclient: monmap: 2024-02-06T21:40:37.645 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:40:37.645 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:37.645 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:40:37.645047+0000 2024-02-06T21:40:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:40:37.645047+0000 2024-02-06T21:40:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:40:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:40:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:40:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:40:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:40:37.646 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:40:37.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.644+0000 7f4036a74640 1 Processor -- start 2024-02-06T21:40:37.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.644+0000 7f4036a74640 1 -- start start 2024-02-06T21:40:37.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.644+0000 7f4036a74640 10 monclient: init 2024-02-06T21:40:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.645+0000 7f4036a74640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.645+0000 7f4036a74640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-02-06T21:40:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.645+0000 7f4036a74640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300b54b0 0x7f40300a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.645+0000 7f4036a74640 10 monclient(hunting): picked mon.noname-b con 0x7f40300b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.645+0000 7f4036a74640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f403013f830 0x7f403013fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.646+0000 7f4036a74640 10 monclient(hunting): picked mon.noname-c con 0x7f403013f830 addr v2:172.21.15.135:3300/0 2024-02-06T21:40:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.646+0000 7f4036a74640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4030142680 addr v1:172.21.15.135:6789/0 2024-02-06T21:40:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.646+0000 7f4036a74640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4030143200 con 0x7f4030142680 2024-02-06T21:40:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.646+0000 7f4036a74640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.646+0000 7f4036a74640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40300ab520 con 0x7f403013f830 2024-02-06T21:40:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.646+0000 7f4036a74640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.646+0000 7f4036a74640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40300b5890 con 0x7f40300b54b0 2024-02-06T21:40:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.646+0000 7f4036a74640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.646+0000 7f4036a74640 10 monclient(hunting): authenticate will time out at 1267.984176s 2024-02-06T21:40:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.646+0000 7f4034810640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300b54b0 0x7f40300a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:37.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.646+0000 7f402ffff640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f403013f830 0x7f403013fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:37.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.647+0000 7f4034810640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300b54b0 0x7f40300a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57100/0 (socket says 172.21.15.120:57100) 2024-02-06T21:40:37.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.647+0000 7f4034810640 1 -- 172.21.15.120:0/2407917380 learned_addr learned my addr 172.21.15.120:0/2407917380 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:40:37.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.647+0000 7f4034810640 10 monclient(hunting): get_auth_request con 0x7f40300b54b0 auth_method 0 2024-02-06T21:40:37.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.647+0000 7f4034810640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:37.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.647+0000 7f4034810640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:37.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.647+0000 7f4034810640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:37.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.647+0000 7f402ffff640 10 monclient(hunting): get_auth_request con 0x7f403013f830 auth_method 0 2024-02-06T21:40:37.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.647+0000 7f402ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:37.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.647+0000 7f402ffff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:37.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.647+0000 7f402ffff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:37.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.648+0000 7f4034810640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:37.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.648+0000 7f4034810640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:37.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.648+0000 7f4034810640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.648+0000 7f402ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.648+0000 7f402ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.648+0000 7f402ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.650+0000 7f4034810640 10 monclient(hunting): handle_auth_done global_id 4142 payload 274 2024-02-06T21:40:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.650+0000 7f4034810640 1 -- 172.21.15.120:0/2407917380 >> v2:172.21.15.135:3300/0 conn(0x7f403013f830 msgr2=0x7f403013fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.650+0000 7f4034810640 1 --2- 172.21.15.120:0/2407917380 >> v2:172.21.15.135:3300/0 conn(0x7f403013f830 0x7f403013fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.650+0000 7f4034810640 1 -- 172.21.15.120:0/2407917380 >> v1:172.21.15.135:6789/0 conn(0x7f4030142680 legacy=0x7f4030142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.651+0000 7f4034810640 10 monclient: _finish_hunting 0 2024-02-06T21:40:37.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.651+0000 7f4034810640 1 monclient: found mon.noname-b 2024-02-06T21:40:37.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.651+0000 7f4034810640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:37.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.651+0000 7f4034810640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:40:37.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.651+0000 7f4034810640 1 -- 172.21.15.120:0/2407917380 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40300ab720 con 0x7f40300b54b0 2024-02-06T21:40:37.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.653+0000 7f4034810640 1 --2- 172.21.15.120:0/2407917380 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300b54b0 0x7f40300a9860 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4024009870 tx=0x7f402402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=607039164da86d86 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:37.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f402f7fe640 1 -- 172.21.15.120:0/2407917380 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4024030840 con 0x7f40300b54b0 2024-02-06T21:40:37.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f402f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:37.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f402f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:40:37.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f402f7fe640 10 monclient: dump: 2024-02-06T21:40:37.663 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:40:37.663 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:37.663 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.663 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.664 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:37.664 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:37.664 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:37.664 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:37.664 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:37.664 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:40:37.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f402f7fe640 10 monclient: _finish_auth 0 2024-02-06T21:40:37.667 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f402f7fe640 10 monclient: _check_auth_tickets 2024-02-06T21:40:37.667 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f402f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:37.668 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f402f7fe640 1 -- 172.21.15.120:0/2407917380 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40240309a0 con 0x7f40300b54b0 2024-02-06T21:40:37.668 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f402f7fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:37.668 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f402f7fe640 1 -- 172.21.15.120:0/2407917380 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4024030d30 con 0x7f40300b54b0 2024-02-06T21:40:37.668 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f402f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:37.668 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f402f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:40:37.668 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f402f7fe640 10 monclient: dump: 2024-02-06T21:40:37.668 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:40:37.668 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:37.668 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.668 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.668 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:37.668 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:37.668 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:37.668 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:37.669 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:37.669 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:40:37.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f4036a74640 5 monclient: authenticate success, global_id 4142 2024-02-06T21:40:37.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f4036a74640 10 monclient: get_monmap_and_config success 2024-02-06T21:40:37.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.658+0000 7f4036a74640 4 set_mon_vals no callback set 2024-02-06T21:40:37.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.659+0000 7f4036a74640 10 monclient: shutdown 2024-02-06T21:40:37.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.659+0000 7f4036a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:37.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.659+0000 7f4036a74640 1 -- 172.21.15.120:0/2407917380 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300b54b0 msgr2=0x7f40300a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:37.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.659+0000 7f4036a74640 1 --2- 172.21.15.120:0/2407917380 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300b54b0 0x7f40300a9860 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4024009870 tx=0x7f402402b980 comp rx=0 tx=0).stop 2024-02-06T21:40:37.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.659+0000 7f4036a74640 1 -- 172.21.15.120:0/2407917380 shutdown_connections 2024-02-06T21:40:37.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.659+0000 7f4036a74640 1 --2- 172.21.15.120:0/2407917380 >> v2:172.21.15.135:3300/0 conn(0x7f403013f830 0x7f403013fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:37.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.659+0000 7f4036a74640 1 --2- 172.21.15.120:0/2407917380 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300b54b0 0x7f40300a9860 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:37.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.659+0000 7f4036a74640 1 -- 172.21.15.120:0/2407917380 >> 172.21.15.120:0/2407917380 conn(0x7f40300b8660 msgr2=0x7f40300b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:37.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.659+0000 7f4036a74640 1 -- 172.21.15.120:0/2407917380 shutdown_connections 2024-02-06T21:40:37.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.659+0000 7f4036a74640 1 -- 172.21.15.120:0/2407917380 wait complete. 2024-02-06T21:40:37.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.660+0000 7f4036a74640 10 monclient: build_initial_monmap 2024-02-06T21:40:37.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.660+0000 7f4036a74640 10 monclient: monmap: 2024-02-06T21:40:37.670 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:40:37.670 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:40:37.670 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:40:37.670 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:40:37.670 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:37.670 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:37.670 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:40:37.670 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:40:37.670 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:40:37.670 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:40:37.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.661+0000 7f4036a74640 1 librados: starting msgr at 2024-02-06T21:40:37.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.661+0000 7f4036a74640 1 librados: starting objecter 2024-02-06T21:40:37.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.661+0000 7f4036a74640 1 Processor -- start 2024-02-06T21:40:37.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.661+0000 7f4036a74640 1 -- start start 2024-02-06T21:40:37.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.661+0000 7f4036a74640 1 librados: setting wanted keys 2024-02-06T21:40:37.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.662+0000 7f4036a74640 1 librados: calling monclient init 2024-02-06T21:40:37.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.662+0000 7f4036a74640 10 monclient: init 2024-02-06T21:40:37.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.663+0000 7f4036a74640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:37.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.663+0000 7f4036a74640 10 monclient: _add_conns ranks=[0,2,1] 2024-02-06T21:40:37.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.663+0000 7f4036a74640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40300b54b0 0x7f4030136a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:37.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.663+0000 7f4036a74640 10 monclient(hunting): picked mon.noname-a con 0x7f40300b54b0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:40:37.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.663+0000 7f4036a74640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403013f830 0x7f4030135ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:37.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.663+0000 7f4036a74640 10 monclient(hunting): picked mon.noname-c con 0x7f403013f830 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:37.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.663+0000 7f4036a74640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4030142680 0x7f4030133990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:37.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.663+0000 7f4036a74640 10 monclient(hunting): picked mon.noname-b con 0x7f4030142680 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:40:37.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.663+0000 7f4036a74640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:37.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.663+0000 7f4036a74640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4030134d70 con 0x7f40300b54b0 2024-02-06T21:40:37.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.663+0000 7f4036a74640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:37.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.663+0000 7f4036a74640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4030135050 con 0x7f4030142680 2024-02-06T21:40:37.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.664+0000 7f4036a74640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:37.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.664+0000 7f4036a74640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4030135330 con 0x7f403013f830 2024-02-06T21:40:37.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.664+0000 7f4036a74640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:37.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.664+0000 7f4036a74640 10 monclient(hunting): authenticate will time out at 1268.001692s 2024-02-06T21:40:37.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.665+0000 7f4035011640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4030142680 0x7f4030133990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:37.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.665+0000 7f4035011640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4030142680 0x7f4030133990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:45468/0 (socket says 172.21.15.120:45468) 2024-02-06T21:40:37.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.665+0000 7f4035011640 1 -- 172.21.15.120:0/333817205 learned_addr learned my addr 172.21.15.120:0/333817205 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:40:37.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.665+0000 7f4035011640 10 monclient(hunting): get_auth_request con 0x7f4030142680 auth_method 0 2024-02-06T21:40:37.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.665+0000 7f4035011640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:37.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.665+0000 7f4035011640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:37.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.665+0000 7f4035011640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:37.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.665+0000 7f402ffff640 1 --2- 172.21.15.120:0/333817205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403013f830 0x7f4030135ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:37.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.665+0000 7f402ffff640 10 monclient(hunting): get_auth_request con 0x7f403013f830 auth_method 0 2024-02-06T21:40:37.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.665+0000 7f402ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:37.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.665+0000 7f402ffff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:37.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.665+0000 7f402ffff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:37.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.665+0000 7f4035011640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:37.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.665+0000 7f4035011640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:37.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.665+0000 7f4035011640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:37.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.666+0000 7f402ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:37.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.666+0000 7f402ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:37.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.667+0000 7f402ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:37.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.670+0000 7f4035011640 10 monclient(hunting): handle_auth_done global_id 4159 payload 824 2024-02-06T21:40:37.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.670+0000 7f4035011640 1 -- 172.21.15.120:0/333817205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403013f830 msgr2=0x7f4030135ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:37.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.670+0000 7f4035011640 1 --2- 172.21.15.120:0/333817205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403013f830 0x7f4030135ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:37.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.670+0000 7f4035011640 1 -- 172.21.15.120:0/333817205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40300b54b0 msgr2=0x7f4030136a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:37.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.670+0000 7f4035011640 1 --2- 172.21.15.120:0/333817205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40300b54b0 0x7f4030136a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:37.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.670+0000 7f4035011640 10 monclient: _finish_hunting 0 2024-02-06T21:40:37.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.670+0000 7f4035011640 1 monclient: found mon.noname-b 2024-02-06T21:40:37.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.670+0000 7f4035011640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:37.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.670+0000 7f4035011640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:40:37.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.670+0000 7f4035011640 1 -- 172.21.15.120:0/333817205 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40240094d0 con 0x7f4030142680 2024-02-06T21:40:37.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.673+0000 7f4035011640 1 --2- 172.21.15.120:0/333817205 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4030142680 0x7f4030133990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f402800c730 tx=0x7f402800cb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:37.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.673+0000 7f402d7fa640 10 client.?.objecter ms_handle_connect 0x7f4030142680 2024-02-06T21:40:37.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.674+0000 7f402d7fa640 10 client.?.objecter resend_mon_ops 2024-02-06T21:40:37.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f402d7fa640 1 -- 172.21.15.120:0/333817205 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028006d50 con 0x7f4030142680 2024-02-06T21:40:37.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f402d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:37.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f402d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:40:37.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f402d7fa640 10 monclient: dump: 2024-02-06T21:40:37.680 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:40:37.680 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:37.680 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.680 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.680 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:37.680 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f402d7fa640 10 monclient: _finish_auth 0 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f402d7fa640 10 monclient: _check_auth_tickets 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f402d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f402d7fa640 1 -- 172.21.15.120:0/333817205 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4028003970 con 0x7f4030142680 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f402d7fa640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f402d7fa640 1 -- 172.21.15.120:0/333817205 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028003d20 con 0x7f4030142680 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f402d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f402d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f402d7fa640 10 monclient: dump: 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.682 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:37.682 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:37.682 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:37.682 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:37.682 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:37.682 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:37.682 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:40:37.684 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f4036a74640 5 monclient: authenticate success, global_id 4159 2024-02-06T21:40:37.684 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f4036a74640 10 monclient: _renew_subs 2024-02-06T21:40:37.684 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f4036a74640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:37.684 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f4036a74640 1 -- 172.21.15.120:0/333817205 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4030133190 con 0x7f4030142680 2024-02-06T21:40:37.684 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f402effd640 4 set_mon_vals no callback set 2024-02-06T21:40:37.684 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f4036a74640 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:40:37.684 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f4036a74640 10 monclient: _renew_subs 2024-02-06T21:40:37.684 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f4036a74640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:37.684 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f4036a74640 1 -- 172.21.15.120:0/333817205 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4030136000 con 0x7f4030142680 2024-02-06T21:40:37.684 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.678+0000 7f4036a74640 1 librados: init done 2024-02-06T21:40:37.684 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.679+0000 7f4036a74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:37.684 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.679+0000 7f4036a74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:37.684 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.679+0000 7f4036a74640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:37.685 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.679+0000 7f4036a74640 1 -- 172.21.15.120:0/333817205 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40301336b0 con 0x7f4030142680 2024-02-06T21:40:37.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.786+0000 7f402d7fa640 1 -- 172.21.15.120:0/333817205 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 2) v1 ==== 44919+0+0 (secure 0 0 0) 0x7f40280050d0 con 0x7f4030142680 2024-02-06T21:40:37.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.794+0000 7f402d7fa640 1 -- 172.21.15.120:0/333817205 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f402800a640 con 0x7f4030142680 2024-02-06T21:40:37.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.794+0000 7f402d7fa640 10 client.4159.objecter ms_dispatch 0x7f4030000b90 osd_map(10..10 src has 1..10) v4 2024-02-06T21:40:37.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.794+0000 7f402d7fa640 3 client.4159.objecter handle_osd_map got epochs [10,10] > 0 2024-02-06T21:40:37.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.794+0000 7f402d7fa640 3 client.4159.objecter handle_osd_map decoding full epoch 10 2024-02-06T21:40:37.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.794+0000 7f402d7fa640 20 client.4159.objecter dump_active .. 0 homeless 2024-02-06T21:40:37.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.794+0000 7f402d7fa640 1 -- 172.21.15.120:0/333817205 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4028023390 con 0x7f4030142680 2024-02-06T21:40:37.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.794+0000 7f402d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:37.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.794+0000 7f402d7fa640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:40:37.920 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.918+0000 7f4036a74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:40:37.920 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.918+0000 7f4036a74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:40:37.920 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.918+0000 7f4036a74640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:37.920 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.918+0000 7f4036a74640 1 -- 172.21.15.120:0/333817205 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40301336b0 con 0x7f4030142680 2024-02-06T21:40:37.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.997+0000 7f402d7fa640 1 -- 172.21.15.120:0/333817205 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f40301336b0 con 0x7f4030142680 2024-02-06T21:40:37.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.997+0000 7f402d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:40:37.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.997+0000 7f402d7fa640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:40:37.999 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:40:37.999 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":10,"fsid":"4e4791bf-0a02-4478-9517-2d2e893f4f7c","created":"2024-02-06T21:40:31.800075+0000","modified":"2024-02-06T21:40:37.354288+0000","last_up_change":"0.000000","last_in_change":"2024-02-06T21:40:37.354288+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4bd0f591-2cf1-42da-8ac1-dce3188580c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81dc801c-3768-4c2c-9adb-6c8300a341f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9289721b-c13f-4a17-be68-3698090f69af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"53534db0-2d4d-44ed-90e8-d67ea878b51a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e4205ab5-787c-44e0-9b96-273b8c9cf1ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b600bc-3afe-44bd-bd3f-690106e7e0ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1d6b9f96-1f9d-44f2-83b0-93f5c342fafb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecae00f5-8e29-47dd-90d5-e51503bc9dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-06T21:40:38.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.998+0000 7f4036a74640 10 librados: watch_flush enter 2024-02-06T21:40:38.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.998+0000 7f4036a74640 10 librados: watch_flush exit 2024-02-06T21:40:38.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.998+0000 7f4036a74640 20 client.4159.objecter shutdown clearing up homeless session... 2024-02-06T21:40:38.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.998+0000 7f4036a74640 10 client.4159.objecter successfully canceled tick 2024-02-06T21:40:38.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.998+0000 7f4036a74640 10 monclient: shutdown 2024-02-06T21:40:38.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.998+0000 7f4036a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:38.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.998+0000 7f4036a74640 1 -- 172.21.15.120:0/333817205 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4030142680 msgr2=0x7f4030133990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:38.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.998+0000 7f4036a74640 1 --2- 172.21.15.120:0/333817205 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4030142680 0x7f4030133990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f402800c730 tx=0x7f402800cb30 comp rx=0 tx=0).stop 2024-02-06T21:40:38.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.999+0000 7f4036a74640 1 -- 172.21.15.120:0/333817205 shutdown_connections 2024-02-06T21:40:38.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.999+0000 7f4036a74640 1 --2- 172.21.15.120:0/333817205 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4030142680 0x7f4030133990 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.999+0000 7f4036a74640 1 --2- 172.21.15.120:0/333817205 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403013f830 0x7f4030135ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.999+0000 7f4036a74640 1 --2- 172.21.15.120:0/333817205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40300b54b0 0x7f4030136a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.999+0000 7f4036a74640 1 -- 172.21.15.120:0/333817205 >> 172.21.15.120:0/333817205 conn(0x7f40300b8660 msgr2=0x7f4030142ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.999+0000 7f4036a74640 1 -- 172.21.15.120:0/333817205 shutdown_connections 2024-02-06T21:40:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.999+0000 7f4036a74640 1 -- 172.21.15.120:0/333817205 wait complete. 2024-02-06T21:40:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:40:37.999+0000 7f4036a74640 1 librados: shutdown 2024-02-06T21:40:38.022 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-02-06T21:40:38.022 INFO:tasks.ceph:Waiting for OSDs to come up 2024-02-06T21:40:38.828 DEBUG:teuthology.orchestra.run.smithi120:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-06T21:40:38.973 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.971+0000 7fd34e874640 10 monclient: get_monmap_and_config 2024-02-06T21:40:38.973 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.971+0000 7fd34e874640 10 monclient: build_initial_monmap 2024-02-06T21:40:38.973 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.971+0000 7fd34e874640 10 monclient: monmap: 2024-02-06T21:40:38.973 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:40:38.973 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:38.973 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:40:38.972836+0000 2024-02-06T21:40:38.973 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:40:38.972836+0000 2024-02-06T21:40:38.973 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:38.974 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:38.974 INFO:teuthology.misc.health.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:40:38.974 INFO:teuthology.misc.health.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:40:38.974 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:40:38.974 INFO:teuthology.misc.health.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:40:38.974 INFO:teuthology.misc.health.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:40:38.974 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:38.976 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.972+0000 7fd34e874640 1 Processor -- start 2024-02-06T21:40:38.976 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.972+0000 7fd34e874640 1 -- start start 2024-02-06T21:40:38.976 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.972+0000 7fd34e874640 10 monclient: init 2024-02-06T21:40:38.977 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.973+0000 7fd34e874640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:38.977 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.973+0000 7fd34e874640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-02-06T21:40:38.977 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.973+0000 7fd34e874640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3480b5060 addr v1:172.21.15.120:6789/0 2024-02-06T21:40:38.977 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.973+0000 7fd34e874640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd34813fe70 0x7fd348142540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:38.977 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.973+0000 7fd34e874640 10 monclient(hunting): picked mon.noname-a con 0x7fd34813fe70 addr v2:172.21.15.120:3300/0 2024-02-06T21:40:38.977 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.973+0000 7fd34e874640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3480a9780 0x7fd348142a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:38.977 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.973+0000 7fd34e874640 10 monclient(hunting): picked mon.noname-b con 0x7fd3480a9780 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:38.977 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.973+0000 7fd34e874640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd348142fc0 con 0x7fd3480b5060 2024-02-06T21:40:38.977 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.973+0000 7fd34e874640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:38.977 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.973+0000 7fd34e874640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3480b5440 con 0x7fd34813fe70 2024-02-06T21:40:38.977 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.973+0000 7fd34e874640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:38.977 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.973+0000 7fd34e874640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3480a9d80 con 0x7fd3480a9780 2024-02-06T21:40:38.977 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.973+0000 7fd34e874640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:38.977 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.973+0000 7fd34e874640 10 monclient(hunting): authenticate will time out at 1269.311441s 2024-02-06T21:40:38.978 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.974+0000 7fd34ce11640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3480a9780 0x7fd348142a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:38.978 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.975+0000 7fd34ce11640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3480a9780 0x7fd348142a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57114/0 (socket says 172.21.15.120:57114) 2024-02-06T21:40:38.978 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.975+0000 7fd34ce11640 1 -- 172.21.15.120:0/3801164156 learned_addr learned my addr 172.21.15.120:0/3801164156 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:40:38.978 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.975+0000 7fd34ce11640 10 monclient(hunting): get_auth_request con 0x7fd3480a9780 auth_method 0 2024-02-06T21:40:38.978 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.975+0000 7fd34ce11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:38.978 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.975+0000 7fd34ce11640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:38.978 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.975+0000 7fd34ce11640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:38.978 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.975+0000 7fd34ce11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:38.979 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.975+0000 7fd34ce11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:38.979 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.975+0000 7fd34ce11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:38.980 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.978+0000 7fd34ce11640 10 monclient(hunting): handle_auth_done global_id 4166 payload 274 2024-02-06T21:40:38.980 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.978+0000 7fd34ce11640 1 -- 172.21.15.120:0/3801164156 >> v2:172.21.15.120:3300/0 conn(0x7fd34813fe70 msgr2=0x7fd348142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:38.980 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.978+0000 7fd34ce11640 1 --2- 172.21.15.120:0/3801164156 >> v2:172.21.15.120:3300/0 conn(0x7fd34813fe70 0x7fd348142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:38.980 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.978+0000 7fd34ce11640 1 -- 172.21.15.120:0/3801164156 >> v1:172.21.15.120:6789/0 conn(0x7fd3480b5060 legacy=0x7fd34813f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:38.980 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.978+0000 7fd34ce11640 10 monclient: _finish_hunting 0 2024-02-06T21:40:38.980 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.978+0000 7fd34ce11640 1 monclient: found mon.noname-b 2024-02-06T21:40:38.980 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.978+0000 7fd34ce11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:38.981 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.978+0000 7fd34ce11640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:40:38.981 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.978+0000 7fd34ce11640 1 -- 172.21.15.120:0/3801164156 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3480ab070 con 0x7fd3480a9780 2024-02-06T21:40:38.987 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.986+0000 7fd34ce11640 1 --2- 172.21.15.120:0/3801164156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3480a9780 0x7fd348142a80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd340009e50 tx=0x7fd34002be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cc0c37a33c0f287 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:38.998 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd346ffd640 1 -- 172.21.15.120:0/3801164156 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34002c9f0 con 0x7fd3480a9780 2024-02-06T21:40:38.998 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd346ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:38.998 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd346ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:40:38.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd346ffd640 10 monclient: dump: 2024-02-06T21:40:38.999 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:40:38.999 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:38.999 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:38.999 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:38.999 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:38.999 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:38.999 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:38.999 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:38.999 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:38.999 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:38.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd346ffd640 10 monclient: _finish_auth 0 2024-02-06T21:40:38.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd346ffd640 10 monclient: _check_auth_tickets 2024-02-06T21:40:38.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd346ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:38.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd346ffd640 1 -- 172.21.15.120:0/3801164156 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34002cb50 con 0x7fd3480a9780 2024-02-06T21:40:39.000 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd346ffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:39.000 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd346ffd640 1 -- 172.21.15.120:0/3801164156 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3400343f0 con 0x7fd3480a9780 2024-02-06T21:40:39.000 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd346ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:39.000 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd346ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:40:39.000 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd346ffd640 10 monclient: dump: 2024-02-06T21:40:39.000 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:40:39.000 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:39.000 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:39.000 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:39.000 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:39.000 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:39.000 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:39.000 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:39.000 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:39.001 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:39.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd34e874640 5 monclient: authenticate success, global_id 4166 2024-02-06T21:40:39.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd34e874640 10 monclient: get_monmap_and_config success 2024-02-06T21:40:39.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.995+0000 7fd34e874640 4 set_mon_vals no callback set 2024-02-06T21:40:39.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.996+0000 7fd34e874640 10 monclient: shutdown 2024-02-06T21:40:39.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.996+0000 7fd34e874640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:39.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.996+0000 7fd34e874640 1 -- 172.21.15.120:0/3801164156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3480a9780 msgr2=0x7fd348142a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:39.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.996+0000 7fd34e874640 1 --2- 172.21.15.120:0/3801164156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3480a9780 0x7fd348142a80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd340009e50 tx=0x7fd34002be50 comp rx=0 tx=0).stop 2024-02-06T21:40:39.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.997+0000 7fd34e874640 1 -- 172.21.15.120:0/3801164156 shutdown_connections 2024-02-06T21:40:39.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.997+0000 7fd34e874640 1 --2- 172.21.15.120:0/3801164156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3480a9780 0x7fd348142a80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:39.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.998+0000 7fd34e874640 1 --2- 172.21.15.120:0/3801164156 >> v2:172.21.15.120:3300/0 conn(0x7fd34813fe70 0x7fd348142540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:39.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.998+0000 7fd34e874640 1 -- 172.21.15.120:0/3801164156 >> 172.21.15.120:0/3801164156 conn(0x7fd3480724d0 msgr2=0x7fd3480728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:39.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.998+0000 7fd34e874640 1 -- 172.21.15.120:0/3801164156 shutdown_connections 2024-02-06T21:40:39.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.998+0000 7fd34e874640 1 -- 172.21.15.120:0/3801164156 wait complete. 2024-02-06T21:40:39.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.999+0000 7fd34e874640 10 monclient: build_initial_monmap 2024-02-06T21:40:39.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:38.999+0000 7fd34e874640 10 monclient: monmap: 2024-02-06T21:40:39.003 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:40:39.003 INFO:teuthology.misc.health.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:40:39.003 INFO:teuthology.misc.health.smithi120.stderr:last_changed 0.000000 2024-02-06T21:40:39.003 INFO:teuthology.misc.health.smithi120.stderr:created 0.000000 2024-02-06T21:40:39.003 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:39.003 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:39.003 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:40:39.003 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:40:39.003 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:40:39.003 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:39.004 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.000+0000 7fd34e874640 1 librados: starting msgr at 2024-02-06T21:40:39.004 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.001+0000 7fd34e874640 1 librados: starting objecter 2024-02-06T21:40:39.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.002+0000 7fd34e874640 1 Processor -- start 2024-02-06T21:40:39.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.002+0000 7fd34e874640 1 -- start start 2024-02-06T21:40:39.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.003+0000 7fd34e874640 1 librados: setting wanted keys 2024-02-06T21:40:39.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.003+0000 7fd34e874640 1 librados: calling monclient init 2024-02-06T21:40:39.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.003+0000 7fd34e874640 10 monclient: init 2024-02-06T21:40:39.007 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.004+0000 7fd34e874640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:39.007 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.004+0000 7fd34e874640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T21:40:39.007 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.005+0000 7fd34e874640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3480a9780 0x7fd348144ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:39.007 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.005+0000 7fd34e874640 10 monclient(hunting): picked mon.noname-b con 0x7fd3480a9780 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:40:39.007 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.005+0000 7fd34e874640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3480b5060 0x7fd3480b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:39.007 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.005+0000 7fd34e874640 10 monclient(hunting): picked mon.noname-a con 0x7fd3480b5060 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:40:39.008 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.005+0000 7fd34e874640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd34813fe70 0x7fd3480b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:39.008 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.005+0000 7fd34e874640 10 monclient(hunting): picked mon.noname-c con 0x7fd34813fe70 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:39.008 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.005+0000 7fd34e874640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:39.008 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.005+0000 7fd34e874640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd348146710 con 0x7fd3480b5060 2024-02-06T21:40:39.008 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.005+0000 7fd34e874640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:39.008 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.005+0000 7fd34e874640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3480b5d10 con 0x7fd3480a9780 2024-02-06T21:40:39.008 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.005+0000 7fd34e874640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:39.008 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.005+0000 7fd34e874640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd348145610 con 0x7fd34813fe70 2024-02-06T21:40:39.008 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.005+0000 7fd34e874640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:39.008 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.005+0000 7fd34e874640 10 monclient(hunting): authenticate will time out at 1269.343403s 2024-02-06T21:40:39.009 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.006+0000 7fd34ce11640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd34813fe70 0x7fd3480b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:39.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.007+0000 7fd34ce11640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd34813fe70 0x7fd3480b6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57130/0 (socket says 172.21.15.120:57130) 2024-02-06T21:40:39.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.007+0000 7fd34ce11640 1 -- 172.21.15.120:0/3315232820 learned_addr learned my addr 172.21.15.120:0/3315232820 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:40:39.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.007+0000 7fd34ce11640 10 monclient(hunting): get_auth_request con 0x7fd34813fe70 auth_method 0 2024-02-06T21:40:39.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.007+0000 7fd34ce11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:39.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.007+0000 7fd34ce11640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:39.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.007+0000 7fd34ce11640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:39.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.008+0000 7fd34ce11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:39.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.008+0000 7fd34ce11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:39.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.008+0000 7fd34ce11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:39.015 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.013+0000 7fd34ce11640 10 monclient(hunting): handle_auth_done global_id 4169 payload 824 2024-02-06T21:40:39.015 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.013+0000 7fd34ce11640 1 -- 172.21.15.120:0/3315232820 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3480a9780 msgr2=0x7fd348144ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:39.015 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.013+0000 7fd34ce11640 1 --2- 172.21.15.120:0/3315232820 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3480a9780 0x7fd348144ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:39.015 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.013+0000 7fd34ce11640 1 -- 172.21.15.120:0/3315232820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3480b5060 msgr2=0x7fd3480b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:39.015 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.013+0000 7fd34ce11640 1 --2- 172.21.15.120:0/3315232820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3480b5060 0x7fd3480b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:39.015 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.013+0000 7fd34ce11640 10 monclient: _finish_hunting 0 2024-02-06T21:40:39.015 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.013+0000 7fd34ce11640 1 monclient: found mon.noname-c 2024-02-06T21:40:39.015 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.013+0000 7fd34ce11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:39.015 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.013+0000 7fd34ce11640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:40:39.015 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.013+0000 7fd34ce11640 1 -- 172.21.15.120:0/3315232820 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd340009b30 con 0x7fd34813fe70 2024-02-06T21:40:39.018 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.017+0000 7fd34ce11640 1 --2- 172.21.15.120:0/3315232820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd34813fe70 0x7fd3480b6a30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd340036040 tx=0x7fd340004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:39.018 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.017+0000 7fd344ff9640 10 client.?.objecter ms_handle_connect 0x7fd34813fe70 2024-02-06T21:40:39.018 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.017+0000 7fd344ff9640 10 client.?.objecter resend_mon_ops 2024-02-06T21:40:39.022 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd344ff9640 1 -- 172.21.15.120:0/3315232820 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34003e450 con 0x7fd34813fe70 2024-02-06T21:40:39.022 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd344ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:39.022 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd344ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:40:39.022 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd344ff9640 10 monclient: dump: 2024-02-06T21:40:39.023 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:40:39.023 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:39.023 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:39.023 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:39.023 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:39.023 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:39.023 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:39.023 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:39.023 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:39.023 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:39.023 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd344ff9640 10 monclient: _finish_auth 0 2024-02-06T21:40:39.023 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd344ff9640 10 monclient: _check_auth_tickets 2024-02-06T21:40:39.023 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd344ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:39.025 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd34e874640 5 monclient: authenticate success, global_id 4169 2024-02-06T21:40:39.025 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd34e874640 10 monclient: _renew_subs 2024-02-06T21:40:39.025 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd34e874640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:39.025 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd34e874640 1 -- 172.21.15.120:0/3315232820 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd348145b30 con 0x7fd34813fe70 2024-02-06T21:40:39.025 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd34e874640 10 client.4169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:40:39.025 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd34e874640 10 monclient: _renew_subs 2024-02-06T21:40:39.025 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd34e874640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:39.025 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd34e874640 1 -- 172.21.15.120:0/3315232820 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd348139b00 con 0x7fd34813fe70 2024-02-06T21:40:39.025 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.021+0000 7fd34e874640 1 librados: init done 2024-02-06T21:40:39.025 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.023+0000 7fd34e874640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:39.025 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.023+0000 7fd34e874640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.023+0000 7fd34e874640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.023+0000 7fd34e874640 1 -- 172.21.15.120:0/3315232820 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3480b6230 con 0x7fd34813fe70 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.023+0000 7fd344ff9640 1 -- 172.21.15.120:0/3315232820 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34003e5b0 con 0x7fd34813fe70 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.023+0000 7fd344ff9640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.023+0000 7fd344ff9640 1 -- 172.21.15.120:0/3315232820 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34003e960 con 0x7fd34813fe70 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.023+0000 7fd344ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.023+0000 7fd344ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.024+0000 7fd344ff9640 10 monclient: dump: 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:39.026 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:39.027 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:39.027 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:39.027 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.024+0000 7fd3467fc640 4 set_mon_vals no callback set 2024-02-06T21:40:39.235 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.234+0000 7fd344ff9640 1 -- 172.21.15.120:0/3315232820 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 2) v1 ==== 44919+0+0 (secure 0 0 0) 0x7fd34003eac0 con 0x7fd34813fe70 2024-02-06T21:40:39.247 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.245+0000 7fd344ff9640 1 -- 172.21.15.120:0/3315232820 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fd340034c50 con 0x7fd34813fe70 2024-02-06T21:40:39.247 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.245+0000 7fd344ff9640 10 client.4169.objecter ms_dispatch 0x7fd348000b90 osd_map(10..10 src has 1..10) v4 2024-02-06T21:40:39.247 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.245+0000 7fd344ff9640 3 client.4169.objecter handle_osd_map got epochs [10,10] > 0 2024-02-06T21:40:39.248 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.245+0000 7fd344ff9640 3 client.4169.objecter handle_osd_map decoding full epoch 10 2024-02-06T21:40:39.248 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.245+0000 7fd344ff9640 20 client.4169.objecter dump_active .. 0 homeless 2024-02-06T21:40:39.248 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.245+0000 7fd344ff9640 1 -- 172.21.15.120:0/3315232820 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd340051430 con 0x7fd34813fe70 2024-02-06T21:40:39.248 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.245+0000 7fd344ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:39.248 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.245+0000 7fd344ff9640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:40:39.266 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:39.266+0000 7f0589610640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-06T21:40:39.372 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.370+0000 7fd34e874640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:40:39.372 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.370+0000 7fd34e874640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:40:39.372 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.370+0000 7fd34e874640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:40:39.372 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.370+0000 7fd34e874640 1 -- 172.21.15.120:0/3315232820 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3480b6230 con 0x7fd34813fe70 2024-02-06T21:40:39.554 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.554+0000 7fd344ff9640 1 -- 172.21.15.120:0/3315232820 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mgrmap(e 3) v1 ==== 89626+0+0 (secure 0 0 0) 0x7fd340050ce0 con 0x7fd34813fe70 2024-02-06T21:40:39.555 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.554+0000 7fd344ff9640 1 --2- 172.21.15.120:0/3315232820 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd33406d6c0 0x7fd33406fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:39.555 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.554+0000 7fd347fff640 1 --2- 172.21.15.120:0/3315232820 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd33406d6c0 0x7fd33406fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:39.555 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.554+0000 7fd347fff640 10 monclient: get_auth_request con 0x7fd33406d6c0 auth_method 0 2024-02-06T21:40:39.555 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.555+0000 7fd344ff9640 10 client.4169.objecter ms_handle_connect 0x7fd33406d6c0 2024-02-06T21:40:39.556 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.555+0000 7fd347fff640 1 --2- 172.21.15.120:0/3315232820 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd33406d6c0 0x7fd33406fab0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd338001560 tx=0x7fd338007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:39.710 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T21:40:39.709+0000 7f0589610640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-02-06T21:40:39.822 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.822+0000 7fd344ff9640 1 -- 172.21.15.120:0/3315232820 <== mon.2 v2:172.21.15.120:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fd340063c20 con 0x7fd34813fe70 2024-02-06T21:40:39.823 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.822+0000 7fd344ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:40:39.823 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.822+0000 7fd344ff9640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:40:39.823 INFO:teuthology.misc.health.smithi120.stdout: 2024-02-06T21:40:39.823 INFO:teuthology.misc.health.smithi120.stdout:{"epoch":10,"fsid":"4e4791bf-0a02-4478-9517-2d2e893f4f7c","created":"2024-02-06T21:40:31.800075+0000","modified":"2024-02-06T21:40:37.354288+0000","last_up_change":"0.000000","last_in_change":"2024-02-06T21:40:37.354288+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4bd0f591-2cf1-42da-8ac1-dce3188580c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81dc801c-3768-4c2c-9adb-6c8300a341f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9289721b-c13f-4a17-be68-3698090f69af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"53534db0-2d4d-44ed-90e8-d67ea878b51a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e4205ab5-787c-44e0-9b96-273b8c9cf1ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b600bc-3afe-44bd-bd3f-690106e7e0ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1d6b9f96-1f9d-44f2-83b0-93f5c342fafb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecae00f5-8e29-47dd-90d5-e51503bc9dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-06T21:40:39.825 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 10 librados: watch_flush enter 2024-02-06T21:40:39.825 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 10 librados: watch_flush exit 2024-02-06T21:40:39.825 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 20 client.4169.objecter shutdown clearing up homeless session... 2024-02-06T21:40:39.825 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 10 client.4169.objecter successfully canceled tick 2024-02-06T21:40:39.825 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 1 -- 172.21.15.120:0/3315232820 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd33406d6c0 msgr2=0x7fd33406fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:39.825 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 1 --2- 172.21.15.120:0/3315232820 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd33406d6c0 0x7fd33406fab0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd338001560 tx=0x7fd338007990 comp rx=0 tx=0).stop 2024-02-06T21:40:39.825 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 10 monclient: shutdown 2024-02-06T21:40:39.826 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:39.826 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 1 -- 172.21.15.120:0/3315232820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd34813fe70 msgr2=0x7fd3480b6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:39.826 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 1 --2- 172.21.15.120:0/3315232820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd34813fe70 0x7fd3480b6a30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd340036040 tx=0x7fd340004a40 comp rx=0 tx=0).stop 2024-02-06T21:40:39.826 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 1 -- 172.21.15.120:0/3315232820 shutdown_connections 2024-02-06T21:40:39.826 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 1 --2- 172.21.15.120:0/3315232820 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd33406d6c0 0x7fd33406fab0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:39.826 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 1 --2- 172.21.15.120:0/3315232820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd34813fe70 0x7fd3480b6a30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:39.826 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 1 --2- 172.21.15.120:0/3315232820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3480b5060 0x7fd3480b64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:39.826 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 1 --2- 172.21.15.120:0/3315232820 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3480a9780 0x7fd348144ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:39.826 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 1 -- 172.21.15.120:0/3315232820 >> 172.21.15.120:0/3315232820 conn(0x7fd3480724d0 msgr2=0x7fd3480b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:39.826 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 1 -- 172.21.15.120:0/3315232820 shutdown_connections 2024-02-06T21:40:39.826 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 1 -- 172.21.15.120:0/3315232820 wait complete. 2024-02-06T21:40:39.826 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:39.824+0000 7fd34e874640 1 librados: shutdown 2024-02-06T21:40:39.850 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-02-06T21:40:45.001 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:07.468 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.063 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:07.529 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.079 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:07.518 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.144 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:07.583 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.153 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:07.619 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.181 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:07.620 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.219 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:07.685 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.219 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:07.666 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.241 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:07.680 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.252 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:07.699 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.280 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:07.727 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.292 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:07.750 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.312 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:07.761 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.344 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:07.791 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.354 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:07.812 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.411 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:07.857 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:45.608 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:40:45.601+0000 5dc3480 -1 Falling back to public interface 2024-02-06T21:40:45.699 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:40:45.692+0000 5dc3480 -1 Falling back to public interface 2024-02-06T21:40:45.783 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:40:45.776+0000 5dc3480 -1 Falling back to public interface 2024-02-06T21:40:45.821 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:40:45.815+0000 5dc3480 -1 Falling back to public interface 2024-02-06T21:40:45.822 INFO:tasks.ceph.osd.7.smithi135.stderr:2024-02-06T21:40:45.811+0000 5dc3480 -1 Falling back to public interface 2024-02-06T21:40:45.830 INFO:tasks.ceph.osd.7.smithi135.stderr:2024-02-06T21:40:45.828+0000 8c75640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6824/0: (98) Address already in use 2024-02-06T21:40:45.858 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:40:45.851+0000 5dc3480 -1 Falling back to public interface 2024-02-06T21:40:45.937 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:40:45.928+0000 5dc3480 -1 Falling back to public interface 2024-02-06T21:40:45.992 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:40:45.984+0000 5dc3480 -1 Falling back to public interface 2024-02-06T21:40:46.074 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:08.540 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.196 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:08.635 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.278 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:08.744 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.312 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:08.751 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.362 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:08.810 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.362 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:08.810 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.438 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:08.904 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.471 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:08.928 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.528 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:08.974 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.571 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:09.010 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.656 DEBUG:teuthology.orchestra.run.smithi120:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-06T21:40:46.662 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:09.129 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.680 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:09.119 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.728 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:09.175 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.748 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.747+0000 7fe0e0e33640 10 monclient: get_monmap_and_config 2024-02-06T21:40:46.748 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.747+0000 7fe0e0e33640 10 monclient: build_initial_monmap 2024-02-06T21:40:46.748 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.747+0000 7fe0e0e33640 10 monclient: monmap: 2024-02-06T21:40:46.748 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:40:46.748 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:46.748 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:40:46.747954+0000 2024-02-06T21:40:46.748 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:40:46.747954+0000 2024-02-06T21:40:46.748 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:46.748 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:46.748 INFO:teuthology.misc.health.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:40:46.749 INFO:teuthology.misc.health.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:40:46.749 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:40:46.749 INFO:teuthology.misc.health.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:40:46.749 INFO:teuthology.misc.health.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:40:46.749 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:46.750 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.748+0000 7fe0e0e33640 1 Processor -- start 2024-02-06T21:40:46.750 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.748+0000 7fe0e0e33640 1 -- start start 2024-02-06T21:40:46.751 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.748+0000 7fe0e0e33640 10 monclient: init 2024-02-06T21:40:46.751 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.748+0000 7fe0e0e33640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:46.751 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.748+0000 7fe0e0e33640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-02-06T21:40:46.751 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.748+0000 7fe0e0e33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0dc0b5060 0x7fe0dc13f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:46.751 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.749+0000 7fe0e0e33640 10 monclient(hunting): picked mon.noname-b con 0x7fe0dc0b5060 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:46.751 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.749+0000 7fe0e0e33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe0dc0a9780 addr v1:172.21.15.120:6789/0 2024-02-06T21:40:46.751 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.749+0000 7fe0e0e33640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe0dc142c50 0x7fe0dc13fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:46.751 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.749+0000 7fe0e0e33640 10 monclient(hunting): picked mon.noname-a con 0x7fe0dc142c50 addr v2:172.21.15.120:3300/0 2024-02-06T21:40:46.751 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.749+0000 7fe0e0e33640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0dc0a9d80 con 0x7fe0dc0a9780 2024-02-06T21:40:46.751 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.749+0000 7fe0e0e33640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:46.751 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.749+0000 7fe0e0e33640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0dc1401e0 con 0x7fe0dc142c50 2024-02-06T21:40:46.751 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.749+0000 7fe0e0e33640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:46.751 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.749+0000 7fe0e0e33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc0b5440 con 0x7fe0dc0b5060 2024-02-06T21:40:46.751 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.749+0000 7fe0e0e33640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:46.752 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.749+0000 7fe0e0e33640 10 monclient(hunting): authenticate will time out at 1277.085592s 2024-02-06T21:40:46.752 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.749+0000 7fe0da59c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0dc0b5060 0x7fe0dc13f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:46.752 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.749+0000 7fe0dad9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe0dc142c50 0x7fe0dc13fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:46.752 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.750+0000 7fe0d9d9b640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fe0dc0a9780 0x7fe0dc142540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:52200/0 (socket says 172.21.15.120:52200) 2024-02-06T21:40:46.752 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.750+0000 7fe0d9d9b640 1 -- 172.21.15.120:0/1662366787 learned_addr learned my addr 172.21.15.120:0/1662366787 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:40:46.752 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.750+0000 7fe0da59c640 10 monclient(hunting): get_auth_request con 0x7fe0dc0b5060 auth_method 0 2024-02-06T21:40:46.752 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.750+0000 7fe0da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:46.752 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.750+0000 7fe0da59c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:46.752 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.750+0000 7fe0da59c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:46.752 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.750+0000 7fe0dad9d640 10 monclient(hunting): get_auth_request con 0x7fe0dc142c50 auth_method 0 2024-02-06T21:40:46.752 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.750+0000 7fe0dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:46.752 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.750+0000 7fe0dad9d640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:46.752 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.750+0000 7fe0dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:46.753 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.750+0000 7fe0da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:46.754 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.750+0000 7fe0da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:46.754 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.750+0000 7fe0da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:46.754 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.751+0000 7fe0dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:46.754 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.751+0000 7fe0dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:46.754 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.751+0000 7fe0dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:46.754 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.753+0000 7fe0da59c640 10 monclient(hunting): handle_auth_done global_id 4181 payload 274 2024-02-06T21:40:46.754 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.753+0000 7fe0da59c640 1 -- 172.21.15.120:0/1662366787 >> v2:172.21.15.120:3300/0 conn(0x7fe0dc142c50 msgr2=0x7fe0dc13fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:46.754 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.753+0000 7fe0da59c640 1 --2- 172.21.15.120:0/1662366787 >> v2:172.21.15.120:3300/0 conn(0x7fe0dc142c50 0x7fe0dc13fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:46.754 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.753+0000 7fe0da59c640 1 -- 172.21.15.120:0/1662366787 >> v1:172.21.15.120:6789/0 conn(0x7fe0dc0a9780 legacy=0x7fe0dc142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:46.754 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.753+0000 7fe0da59c640 10 monclient: _finish_hunting 0 2024-02-06T21:40:46.754 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.753+0000 7fe0da59c640 1 monclient: found mon.noname-b 2024-02-06T21:40:46.754 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.753+0000 7fe0da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:46.754 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.753+0000 7fe0da59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:40:46.755 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.753+0000 7fe0da59c640 1 -- 172.21.15.120:0/1662366787 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0dc0ab070 con 0x7fe0dc0b5060 2024-02-06T21:40:46.755 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.755+0000 7fe0da59c640 1 --2- 172.21.15.120:0/1662366787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0dc0b5060 0x7fe0dc13f760 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc0097a0 tx=0x7fe0cc02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91d63eddca1c274f server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:46.760 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0d959a640 1 -- 172.21.15.120:0/1662366787 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc031780 con 0x7fe0dc0b5060 2024-02-06T21:40:46.760 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:46.761 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0d959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:40:46.761 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0d959a640 10 monclient: dump: 2024-02-06T21:40:46.761 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:40:46.761 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:46.761 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:46.761 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:46.761 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:46.761 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:46.761 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:46.761 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:46.761 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:46.761 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:46.761 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0d959a640 10 monclient: _finish_auth 0 2024-02-06T21:40:46.762 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0d959a640 10 monclient: _check_auth_tickets 2024-02-06T21:40:46.762 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:46.762 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0d959a640 1 -- 172.21.15.120:0/1662366787 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0cc0318e0 con 0x7fe0dc0b5060 2024-02-06T21:40:46.762 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0e0e33640 5 monclient: authenticate success, global_id 4181 2024-02-06T21:40:46.762 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0e0e33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:40:46.762 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0d959a640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:46.762 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0e0e33640 10 monclient: get_monmap_and_config success 2024-02-06T21:40:46.762 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0e0e33640 4 set_mon_vals no callback set 2024-02-06T21:40:46.762 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0d959a640 1 -- 172.21.15.120:0/1662366787 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc031c70 con 0x7fe0dc0b5060 2024-02-06T21:40:46.762 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:46.762 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0d959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:40:46.762 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.759+0000 7fe0d959a640 10 monclient: dump: 2024-02-06T21:40:46.762 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:40:46.762 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:46.762 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:46.763 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:46.763 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:46.763 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:46.763 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:46.763 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:46.763 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:46.763 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:46.763 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:09.208 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.767 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.760+0000 7fe0e0e33640 10 monclient: shutdown 2024-02-06T21:40:46.767 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.760+0000 7fe0e0e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:46.767 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.760+0000 7fe0e0e33640 1 -- 172.21.15.120:0/1662366787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0dc0b5060 msgr2=0x7fe0dc13f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:46.767 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.760+0000 7fe0e0e33640 1 --2- 172.21.15.120:0/1662366787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0dc0b5060 0x7fe0dc13f760 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc0097a0 tx=0x7fe0cc02b820 comp rx=0 tx=0).stop 2024-02-06T21:40:46.767 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.760+0000 7fe0e0e33640 1 -- 172.21.15.120:0/1662366787 shutdown_connections 2024-02-06T21:40:46.767 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.760+0000 7fe0e0e33640 1 --2- 172.21.15.120:0/1662366787 >> v2:172.21.15.120:3300/0 conn(0x7fe0dc142c50 0x7fe0dc13fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:46.767 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.760+0000 7fe0e0e33640 1 --2- 172.21.15.120:0/1662366787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0dc0b5060 0x7fe0dc13f760 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:46.767 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.760+0000 7fe0e0e33640 1 -- 172.21.15.120:0/1662366787 >> 172.21.15.120:0/1662366787 conn(0x7fe0dc0724d0 msgr2=0x7fe0dc0728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:46.767 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.760+0000 7fe0e0e33640 1 -- 172.21.15.120:0/1662366787 shutdown_connections 2024-02-06T21:40:46.767 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.760+0000 7fe0e0e33640 1 -- 172.21.15.120:0/1662366787 wait complete. 2024-02-06T21:40:46.767 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.761+0000 7fe0e0e33640 10 monclient: build_initial_monmap 2024-02-06T21:40:46.767 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.761+0000 7fe0e0e33640 10 monclient: monmap: 2024-02-06T21:40:46.768 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:40:46.768 INFO:teuthology.misc.health.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:40:46.768 INFO:teuthology.misc.health.smithi120.stderr:last_changed 0.000000 2024-02-06T21:40:46.768 INFO:teuthology.misc.health.smithi120.stderr:created 0.000000 2024-02-06T21:40:46.768 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:46.768 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:46.768 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:40:46.768 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:40:46.768 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:40:46.768 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:46.768 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.762+0000 7fe0e0e33640 1 librados: starting msgr at 2024-02-06T21:40:46.768 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.762+0000 7fe0e0e33640 1 librados: starting objecter 2024-02-06T21:40:46.768 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.762+0000 7fe0e0e33640 1 Processor -- start 2024-02-06T21:40:46.768 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.762+0000 7fe0e0e33640 1 -- start start 2024-02-06T21:40:46.769 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.762+0000 7fe0e0e33640 1 librados: setting wanted keys 2024-02-06T21:40:46.769 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.762+0000 7fe0e0e33640 1 librados: calling monclient init 2024-02-06T21:40:46.769 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.762+0000 7fe0e0e33640 10 monclient: init 2024-02-06T21:40:46.769 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:46.769 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 10 monclient: _add_conns ranks=[2,0,1] 2024-02-06T21:40:46.769 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0dc0a9780 0x7fe0dc138710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:46.769 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 10 monclient(hunting): picked mon.noname-c con 0x7fe0dc0a9780 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:46.769 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0dc0b5060 0x7fe0dc138c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:46.769 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 10 monclient(hunting): picked mon.noname-a con 0x7fe0dc0b5060 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:40:46.769 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0dc142c50 0x7fe0dc1407b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:46.769 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 10 monclient(hunting): picked mon.noname-b con 0x7fe0dc142c50 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:40:46.769 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:46.769 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc140340 con 0x7fe0dc0b5060 2024-02-06T21:40:46.769 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:46.769 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc141b90 con 0x7fe0dc142c50 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc139a30 con 0x7fe0dc0a9780 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0e0e33640 10 monclient(hunting): authenticate will time out at 1277.100087s 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.763+0000 7fe0da59c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0dc0a9780 0x7fe0dc138710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0dad9d640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0dc142c50 0x7fe0dc1407b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0d9d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0dc0b5060 0x7fe0dc138c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0dad9d640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0dc142c50 0x7fe0dc1407b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:45120/0 (socket says 172.21.15.120:45120) 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0dad9d640 1 -- 172.21.15.120:0/4059911032 learned_addr learned my addr 172.21.15.120:0/4059911032 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0dad9d640 10 monclient(hunting): get_auth_request con 0x7fe0dc142c50 auth_method 0 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0dad9d640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0d9d9b640 10 monclient(hunting): get_auth_request con 0x7fe0dc0b5060 auth_method 0 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:46.770 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0d9d9b640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:46.771 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:46.771 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0da59c640 10 monclient(hunting): get_auth_request con 0x7fe0dc0a9780 auth_method 0 2024-02-06T21:40:46.771 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:46.771 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0da59c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:46.771 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0da59c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:46.771 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:46.771 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:46.771 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.764+0000 7fe0dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:46.771 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.765+0000 7fe0da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:46.771 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.765+0000 7fe0da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:46.771 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.765+0000 7fe0da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:46.771 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.765+0000 7fe0d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:46.771 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.765+0000 7fe0d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:46.771 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.765+0000 7fe0d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:46.774 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.767+0000 7fe0dad9d640 10 monclient(hunting): handle_auth_done global_id 4195 payload 824 2024-02-06T21:40:46.774 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.767+0000 7fe0dad9d640 1 -- 172.21.15.120:0/4059911032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0dc0a9780 msgr2=0x7fe0dc138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:46.774 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.767+0000 7fe0dad9d640 1 --2- 172.21.15.120:0/4059911032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0dc0a9780 0x7fe0dc138710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:46.774 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.767+0000 7fe0dad9d640 1 -- 172.21.15.120:0/4059911032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0dc0b5060 msgr2=0x7fe0dc138c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:46.774 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.767+0000 7fe0dad9d640 1 --2- 172.21.15.120:0/4059911032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0dc0b5060 0x7fe0dc138c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:46.774 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.767+0000 7fe0dad9d640 10 monclient: _finish_hunting 0 2024-02-06T21:40:46.774 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.767+0000 7fe0dad9d640 1 monclient: found mon.noname-b 2024-02-06T21:40:46.775 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.767+0000 7fe0dad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:46.775 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.767+0000 7fe0dad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:40:46.775 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.767+0000 7fe0dad9d640 1 -- 172.21.15.120:0/4059911032 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0cc009400 con 0x7fe0dc142c50 2024-02-06T21:40:46.775 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.769+0000 7fe0dad9d640 1 --2- 172.21.15.120:0/4059911032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0dc142c50 0x7fe0dc1407b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400efc0 tx=0x7fe0d400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:46.775 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.769+0000 7fe0caffd640 10 client.?.objecter ms_handle_connect 0x7fe0dc142c50 2024-02-06T21:40:46.775 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.769+0000 7fe0caffd640 10 client.?.objecter resend_mon_ops 2024-02-06T21:40:46.775 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0caffd640 1 -- 172.21.15.120:0/4059911032 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4009ba0 con 0x7fe0dc142c50 2024-02-06T21:40:46.775 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:46.775 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0caffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:40:46.775 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0caffd640 10 monclient: dump: 2024-02-06T21:40:46.775 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:40:46.775 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:46.775 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:46.775 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:46.776 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:46.776 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:46.776 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:46.776 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:46.776 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:46.776 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:46.776 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0caffd640 10 monclient: _finish_auth 0 2024-02-06T21:40:46.776 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0caffd640 10 monclient: _check_auth_tickets 2024-02-06T21:40:46.776 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:46.776 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0caffd640 1 -- 172.21.15.120:0/4059911032 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d400f040 con 0x7fe0dc142c50 2024-02-06T21:40:46.776 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0e0e33640 5 monclient: authenticate success, global_id 4195 2024-02-06T21:40:46.776 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0e0e33640 10 monclient: _renew_subs 2024-02-06T21:40:46.776 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0e0e33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:46.776 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0e0e33640 1 -- 172.21.15.120:0/4059911032 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0dc137990 con 0x7fe0dc142c50 2024-02-06T21:40:46.777 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0caffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:46.777 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0caffd640 1 -- 172.21.15.120:0/4059911032 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d40146c0 con 0x7fe0dc142c50 2024-02-06T21:40:46.777 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:46.777 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:40:46.777 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0caffd640 10 monclient: dump: 2024-02-06T21:40:46.777 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:40:46.777 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:46.777 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:46.777 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:46.777 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:46.777 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:46.777 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:46.777 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:46.777 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:46.777 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:46.778 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0d8d99640 4 set_mon_vals no callback set 2024-02-06T21:40:46.778 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0e0e33640 10 client.4195.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:40:46.778 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0e0e33640 10 monclient: _renew_subs 2024-02-06T21:40:46.778 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0e0e33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:46.778 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0e0e33640 1 -- 172.21.15.120:0/4059911032 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0dc140ed0 con 0x7fe0dc142c50 2024-02-06T21:40:46.778 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.773+0000 7fe0e0e33640 1 librados: init done 2024-02-06T21:40:46.778 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.774+0000 7fe0e0e33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:46.778 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.774+0000 7fe0e0e33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:46.779 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.774+0000 7fe0e0e33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:46.779 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.774+0000 7fe0e0e33640 1 -- 172.21.15.120:0/4059911032 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0dc0aa1f0 con 0x7fe0dc142c50 2024-02-06T21:40:46.792 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.791+0000 7fe0caffd640 1 -- 172.21.15.120:0/4059911032 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 4) v1 ==== 89732+0+0 (secure 0 0 0) 0x7fe0d4014820 con 0x7fe0dc142c50 2024-02-06T21:40:46.792 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.792+0000 7fe0caffd640 1 --2- 172.21.15.120:0/4059911032 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe0bc06d110 0x7fe0bc06f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:46.792 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.792+0000 7fe0da59c640 1 --2- 172.21.15.120:0/4059911032 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe0bc06d110 0x7fe0bc06f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:46.792 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.792+0000 7fe0da59c640 10 monclient: get_auth_request con 0x7fe0bc06d110 auth_method 0 2024-02-06T21:40:46.793 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.792+0000 7fe0da59c640 1 --2- 172.21.15.120:0/4059911032 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe0bc06d110 0x7fe0bc06f5d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc0097a0 tx=0x7fe0cc03d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:46.793 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.792+0000 7fe0caffd640 10 client.4195.objecter ms_handle_connect 0x7fe0bc06d110 2024-02-06T21:40:46.794 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.793+0000 7fe0caffd640 1 -- 172.21.15.120:0/4059911032 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fe0d4059ac0 con 0x7fe0dc142c50 2024-02-06T21:40:46.794 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.794+0000 7fe0caffd640 10 client.4195.objecter ms_dispatch 0x7fe0dc000b90 osd_map(10..10 src has 1..10) v4 2024-02-06T21:40:46.794 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.794+0000 7fe0caffd640 3 client.4195.objecter handle_osd_map got epochs [10,10] > 0 2024-02-06T21:40:46.794 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.794+0000 7fe0caffd640 3 client.4195.objecter handle_osd_map decoding full epoch 10 2024-02-06T21:40:46.794 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:46.794+0000 7fe0caffd640 20 client.4195.objecter dump_active .. 0 homeless 2024-02-06T21:40:46.795 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:09.261 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.842 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:09.300 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.906 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:09.353 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:46.956 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:09.395 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.016 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:09.455 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.017 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:09.484 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.039 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.039+0000 7fe0caffd640 1 -- 172.21.15.120:0/4059911032 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe0d4055d60 con 0x7fe0dc142c50 2024-02-06T21:40:47.039 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.039+0000 7fe0caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:47.040 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.039+0000 7fe0caffd640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:40:47.063 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:09.511 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.086 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:09.552 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.134 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:09.582 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.157 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:09.615 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.237 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.236+0000 7fe0e0e33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:40:47.237 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.236+0000 7fe0e0e33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:40:47.237 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.236+0000 7fe0e0e33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:40:47.237 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.236+0000 7fe0e0e33640 1 -- 172.21.15.120:0/4059911032 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0dc0aa1f0 con 0x7fe0dc142c50 2024-02-06T21:40:47.246 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:09.685 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.254 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.253+0000 7fe0caffd640 1 -- 172.21.15.120:0/4059911032 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fe0dc0aa1f0 con 0x7fe0dc142c50 2024-02-06T21:40:47.254 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.253+0000 7fe0caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:40:47.254 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.253+0000 7fe0caffd640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:40:47.254 INFO:teuthology.misc.health.smithi120.stdout: 2024-02-06T21:40:47.254 INFO:teuthology.misc.health.smithi120.stdout:{"epoch":10,"fsid":"4e4791bf-0a02-4478-9517-2d2e893f4f7c","created":"2024-02-06T21:40:31.800075+0000","modified":"2024-02-06T21:40:37.354288+0000","last_up_change":"0.000000","last_in_change":"2024-02-06T21:40:37.354288+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4bd0f591-2cf1-42da-8ac1-dce3188580c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81dc801c-3768-4c2c-9adb-6c8300a341f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9289721b-c13f-4a17-be68-3698090f69af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"53534db0-2d4d-44ed-90e8-d67ea878b51a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e4205ab5-787c-44e0-9b96-273b8c9cf1ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b600bc-3afe-44bd-bd3f-690106e7e0ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1d6b9f96-1f9d-44f2-83b0-93f5c342fafb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecae00f5-8e29-47dd-90d5-e51503bc9dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-06T21:40:47.257 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.255+0000 7fe0e0e33640 10 librados: watch_flush enter 2024-02-06T21:40:47.257 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.255+0000 7fe0e0e33640 10 librados: watch_flush exit 2024-02-06T21:40:47.257 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.255+0000 7fe0e0e33640 20 client.4195.objecter shutdown clearing up homeless session... 2024-02-06T21:40:47.257 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.255+0000 7fe0e0e33640 10 client.4195.objecter successfully canceled tick 2024-02-06T21:40:47.257 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.256+0000 7fe0e0e33640 1 -- 172.21.15.120:0/4059911032 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe0bc06d110 msgr2=0x7fe0bc06f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:47.257 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.256+0000 7fe0e0e33640 1 --2- 172.21.15.120:0/4059911032 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe0bc06d110 0x7fe0bc06f5d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc0097a0 tx=0x7fe0cc03d040 comp rx=0 tx=0).stop 2024-02-06T21:40:47.257 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.256+0000 7fe0e0e33640 10 monclient: shutdown 2024-02-06T21:40:47.257 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.256+0000 7fe0e0e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:47.258 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.256+0000 7fe0e0e33640 1 -- 172.21.15.120:0/4059911032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0dc142c50 msgr2=0x7fe0dc1407b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:47.258 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.256+0000 7fe0e0e33640 1 --2- 172.21.15.120:0/4059911032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0dc142c50 0x7fe0dc1407b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400efc0 tx=0x7fe0d400c490 comp rx=0 tx=0).stop 2024-02-06T21:40:47.258 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.256+0000 7fe0e0e33640 1 -- 172.21.15.120:0/4059911032 shutdown_connections 2024-02-06T21:40:47.258 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.256+0000 7fe0e0e33640 1 --2- 172.21.15.120:0/4059911032 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe0bc06d110 0x7fe0bc06f5d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:47.258 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.256+0000 7fe0e0e33640 1 --2- 172.21.15.120:0/4059911032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0dc142c50 0x7fe0dc1407b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:47.258 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.256+0000 7fe0e0e33640 1 --2- 172.21.15.120:0/4059911032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0dc0b5060 0x7fe0dc138c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:47.258 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.256+0000 7fe0e0e33640 1 --2- 172.21.15.120:0/4059911032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0dc0a9780 0x7fe0dc138710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:47.258 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.256+0000 7fe0e0e33640 1 -- 172.21.15.120:0/4059911032 >> 172.21.15.120:0/4059911032 conn(0x7fe0dc0724d0 msgr2=0x7fe0dc0728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:47.258 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.256+0000 7fe0e0e33640 1 -- 172.21.15.120:0/4059911032 shutdown_connections 2024-02-06T21:40:47.258 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.256+0000 7fe0e0e33640 1 -- 172.21.15.120:0/4059911032 wait complete. 2024-02-06T21:40:47.258 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:47.256+0000 7fe0e0e33640 1 librados: shutdown 2024-02-06T21:40:47.261 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:09.708 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.266 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-02-06T21:40:47.289 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:09.728 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.308 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:09.774 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.353 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:09.801 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.373 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:09.839 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.427 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:09.876 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.430 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:09.888 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.545 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:09.984 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.549 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:09.996 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.558 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:09.997 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.596 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:10.062 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.641 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:10.088 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.661 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:10.127 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.702 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:10.160 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.740 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:10.189 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.826 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:10.272 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.826 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:10.264 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.842 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:10.281 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.886 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:10.352 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.928 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:10.376 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.952 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:10.418 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:47.973 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:10.431 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.029 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:10.477 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.101 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:10.540 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.116 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:10.563 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.130 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:10.570 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.174 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:10.640 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.215 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:10.662 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.237 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:10.694 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.239 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:10.705 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.321 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:10.770 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.367 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:10.806 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.405 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:10.852 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.423 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:10.862 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.464 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:10.930 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.502 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:10.960 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.507 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:10.954 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.526 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:10.992 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.608 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:11.056 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.634 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:11.073 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.673 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:11.120 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.710 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:11.149 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.736 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:11.202 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.770 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:11.227 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.796 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:11.243 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.895 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:11.344 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:48.960 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:11.407 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.016 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:11.483 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.072 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:11.538 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.143 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:11.582 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.198 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:11.637 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.273 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:11.731 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.291 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:11.758 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.294 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:11.733 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.327 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:11.785 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.346 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:11.812 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.348 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:11.788 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.355 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:11.803 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.437 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:11.884 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.492 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:11.940 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.513 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:11.959 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.550 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:11.998 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:49.567 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:12.014 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:54.073 DEBUG:teuthology.orchestra.run.smithi120:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-06T21:40:54.185 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.183+0000 7f343036c640 10 monclient: get_monmap_and_config 2024-02-06T21:40:54.185 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.183+0000 7f343036c640 10 monclient: build_initial_monmap 2024-02-06T21:40:54.186 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.183+0000 7f343036c640 10 monclient: monmap: 2024-02-06T21:40:54.186 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:40:54.186 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:54.186 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:40:54.185223+0000 2024-02-06T21:40:54.186 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:40:54.185223+0000 2024-02-06T21:40:54.186 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:54.186 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:54.186 INFO:teuthology.misc.health.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:40:54.186 INFO:teuthology.misc.health.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:40:54.186 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:40:54.186 INFO:teuthology.misc.health.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:40:54.186 INFO:teuthology.misc.health.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:40:54.187 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:54.187 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.184+0000 7f343036c640 1 Processor -- start 2024-02-06T21:40:54.187 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.184+0000 7f343036c640 1 -- start start 2024-02-06T21:40:54.188 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.184+0000 7f343036c640 10 monclient: init 2024-02-06T21:40:54.188 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.185+0000 7f343036c640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:54.188 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.186+0000 7f343036c640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-02-06T21:40:54.188 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.186+0000 7f343036c640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f34280b5060 0x7f342813f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:54.188 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.186+0000 7f343036c640 10 monclient(hunting): picked mon.noname-c con 0x7f34280b5060 addr v2:172.21.15.135:3300/0 2024-02-06T21:40:54.188 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.186+0000 7f343036c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34280a9780 addr v1:172.21.15.120:6789/0 2024-02-06T21:40:54.188 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.186+0000 7f343036c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3428142c50 0x7f342813fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:54.188 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.186+0000 7f343036c640 10 monclient(hunting): picked mon.noname-a con 0x7f3428142c50 addr v2:172.21.15.120:3300/0 2024-02-06T21:40:54.188 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.186+0000 7f343036c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34280a9d80 con 0x7f34280a9780 2024-02-06T21:40:54.188 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.186+0000 7f343036c640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:54.189 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.186+0000 7f343036c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34281401e0 con 0x7f3428142c50 2024-02-06T21:40:54.189 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.186+0000 7f343036c640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:54.189 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.186+0000 7f343036c640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34280b5440 con 0x7f34280b5060 2024-02-06T21:40:54.189 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.186+0000 7f343036c640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:54.189 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.186+0000 7f343036c640 10 monclient(hunting): authenticate will time out at 1284.523680s 2024-02-06T21:40:54.196 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.194+0000 7f342e909640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3428142c50 0x7f342813fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:54.196 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.194+0000 7f342d907640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f34280a9780 0x7f3428142540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:48000/0 (socket says 172.21.15.120:48000) 2024-02-06T21:40:54.197 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.194+0000 7f342d907640 1 -- 172.21.15.120:0/590019903 learned_addr learned my addr 172.21.15.120:0/590019903 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:40:54.197 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.195+0000 7f342e909640 10 monclient(hunting): get_auth_request con 0x7f3428142c50 auth_method 0 2024-02-06T21:40:54.197 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.195+0000 7f342e909640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:54.197 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.195+0000 7f342e909640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:54.197 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.195+0000 7f342e909640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:54.198 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.195+0000 7f342e909640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:54.198 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.195+0000 7f342e909640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:54.198 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.195+0000 7f342e909640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:54.199 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.197+0000 7f342e909640 10 monclient(hunting): handle_auth_done global_id 4200 payload 274 2024-02-06T21:40:54.199 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.197+0000 7f342e909640 1 -- 172.21.15.120:0/590019903 >> v2:172.21.15.135:3300/0 conn(0x7f34280b5060 msgr2=0x7f342813f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:40:54.199 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.197+0000 7f342e909640 1 --2- 172.21.15.120:0/590019903 >> v2:172.21.15.135:3300/0 conn(0x7f34280b5060 0x7f342813f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:54.199 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.197+0000 7f342e909640 1 -- 172.21.15.120:0/590019903 >> v1:172.21.15.120:6789/0 conn(0x7f34280a9780 legacy=0x7f3428142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:54.199 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.197+0000 7f342e909640 10 monclient: _finish_hunting 0 2024-02-06T21:40:54.199 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.197+0000 7f342e909640 1 monclient: found mon.noname-a 2024-02-06T21:40:54.199 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.197+0000 7f342e909640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:54.199 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.197+0000 7f342e909640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:40:54.199 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.197+0000 7f342e909640 1 -- 172.21.15.120:0/590019903 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34280ab070 con 0x7f3428142c50 2024-02-06T21:40:54.201 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.200+0000 7f342e909640 1 --2- 172.21.15.120:0/590019903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3428142c50 0x7f342813fca0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f34280732a0 tx=0x7f342002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f408a560f73a8cf server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:54.205 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f342d106640 1 -- 172.21.15.120:0/590019903 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342002cbe0 con 0x7f3428142c50 2024-02-06T21:40:54.206 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f342d106640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:54.206 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f342d106640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:40:54.206 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f342d106640 10 monclient: dump: 2024-02-06T21:40:54.206 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:40:54.206 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:54.206 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:54.206 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:54.206 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:54.206 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:54.206 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:54.206 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:54.206 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:54.207 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:54.207 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f342d106640 10 monclient: _finish_auth 0 2024-02-06T21:40:54.207 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f342d106640 10 monclient: _check_auth_tickets 2024-02-06T21:40:54.207 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f342d106640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:54.207 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f342d106640 1 -- 172.21.15.120:0/590019903 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342002cd40 con 0x7f3428142c50 2024-02-06T21:40:54.207 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f342d106640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:54.207 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f342d106640 1 -- 172.21.15.120:0/590019903 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3420034660 con 0x7f3428142c50 2024-02-06T21:40:54.208 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f342d106640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:54.208 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f342d106640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:40:54.208 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f342d106640 10 monclient: dump: 2024-02-06T21:40:54.208 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:40:54.208 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:54.208 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:54.208 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:54.208 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:54.208 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:54.208 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:54.208 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:54.208 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:54.209 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:54.209 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f343036c640 5 monclient: authenticate success, global_id 4200 2024-02-06T21:40:54.209 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f343036c640 10 monclient: get_monmap_and_config success 2024-02-06T21:40:54.209 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.204+0000 7f343036c640 4 set_mon_vals no callback set 2024-02-06T21:40:54.212 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.205+0000 7f343036c640 10 monclient: shutdown 2024-02-06T21:40:54.212 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.205+0000 7f343036c640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:54.212 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.205+0000 7f343036c640 1 -- 172.21.15.120:0/590019903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3428142c50 msgr2=0x7f342813fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:54.212 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.205+0000 7f343036c640 1 --2- 172.21.15.120:0/590019903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3428142c50 0x7f342813fca0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f34280732a0 tx=0x7f342002be30 comp rx=0 tx=0).stop 2024-02-06T21:40:54.212 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.205+0000 7f343036c640 1 -- 172.21.15.120:0/590019903 shutdown_connections 2024-02-06T21:40:54.212 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.205+0000 7f343036c640 1 --2- 172.21.15.120:0/590019903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3428142c50 0x7f342813fca0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:54.212 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.205+0000 7f343036c640 1 --2- 172.21.15.120:0/590019903 >> v2:172.21.15.135:3300/0 conn(0x7f34280b5060 0x7f342813f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:54.212 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.205+0000 7f343036c640 1 -- 172.21.15.120:0/590019903 >> 172.21.15.120:0/590019903 conn(0x7f34280724d0 msgr2=0x7f34280728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:54.213 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.205+0000 7f343036c640 1 -- 172.21.15.120:0/590019903 shutdown_connections 2024-02-06T21:40:54.213 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.205+0000 7f343036c640 1 -- 172.21.15.120:0/590019903 wait complete. 2024-02-06T21:40:54.213 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.206+0000 7f343036c640 10 monclient: build_initial_monmap 2024-02-06T21:40:54.213 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.206+0000 7f343036c640 10 monclient: monmap: 2024-02-06T21:40:54.213 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:40:54.213 INFO:teuthology.misc.health.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:40:54.213 INFO:teuthology.misc.health.smithi120.stderr:last_changed 0.000000 2024-02-06T21:40:54.213 INFO:teuthology.misc.health.smithi120.stderr:created 0.000000 2024-02-06T21:40:54.213 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:40:54.213 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:54.213 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:40:54.214 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:40:54.214 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:40:54.214 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:54.214 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.206+0000 7f343036c640 1 librados: starting msgr at 2024-02-06T21:40:54.214 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.206+0000 7f343036c640 1 librados: starting objecter 2024-02-06T21:40:54.214 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.206+0000 7f343036c640 1 Processor -- start 2024-02-06T21:40:54.214 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.207+0000 7f343036c640 1 -- start start 2024-02-06T21:40:54.214 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.207+0000 7f343036c640 1 librados: setting wanted keys 2024-02-06T21:40:54.214 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.207+0000 7f343036c640 1 librados: calling monclient init 2024-02-06T21:40:54.214 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.207+0000 7f343036c640 10 monclient: init 2024-02-06T21:40:54.214 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.207+0000 7f343036c640 10 monclient: _reopen_session rank -1 2024-02-06T21:40:54.214 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.207+0000 7f343036c640 10 monclient: _add_conns ranks=[0,2,1] 2024-02-06T21:40:54.214 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.207+0000 7f343036c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34280a9780 0x7f34281db6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.207+0000 7f343036c640 10 monclient(hunting): picked mon.noname-a con 0x7f34280a9780 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.207+0000 7f343036c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34280b5060 0x7f3428141a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f343036c640 10 monclient(hunting): picked mon.noname-c con 0x7f34280b5060 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f343036c640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3428142c50 0x7f3428138be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f343036c640 10 monclient(hunting): picked mon.noname-b con 0x7f3428142c50 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f343036c640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f343036c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3428138290 con 0x7f34280a9780 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f343036c640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f343036c640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34281dbc30 con 0x7f3428142c50 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f343036c640 10 monclient(hunting): start opening mon connection 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f343036c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34281384e0 con 0x7f34280b5060 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f343036c640 10 monclient(hunting): _renew_subs 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f343036c640 10 monclient(hunting): authenticate will time out at 1284.545640s 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f342e108640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34280a9780 0x7f34281db6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f342e108640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34280a9780 0x7f34281db6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:59818/0 (socket says 172.21.15.120:59818) 2024-02-06T21:40:54.215 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f342e108640 1 -- 172.21.15.120:0/230530991 learned_addr learned my addr 172.21.15.120:0/230530991 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:40:54.216 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f342e108640 10 monclient(hunting): get_auth_request con 0x7f34280a9780 auth_method 0 2024-02-06T21:40:54.216 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f342e108640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:54.216 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f342e108640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:54.216 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.208+0000 7f342e108640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:54.216 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.209+0000 7f342d907640 1 --2- 172.21.15.120:0/230530991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34280b5060 0x7f3428141a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:54.216 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.209+0000 7f342e108640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:54.216 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.209+0000 7f342e108640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:54.216 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.209+0000 7f342e108640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:54.216 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.209+0000 7f342d907640 10 monclient(hunting): get_auth_request con 0x7f34280b5060 auth_method 0 2024-02-06T21:40:54.216 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.209+0000 7f342d907640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:40:54.216 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.209+0000 7f342d907640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:40:54.216 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.209+0000 7f342d907640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:40:54.216 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.210+0000 7f342d907640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:40:54.216 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.210+0000 7f342d907640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:40:54.217 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.210+0000 7f342d907640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:40:54.218 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.212+0000 7f342e108640 10 monclient(hunting): handle_auth_done global_id 4203 payload 824 2024-02-06T21:40:54.218 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.212+0000 7f342e108640 1 -- 172.21.15.120:0/230530991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34280b5060 msgr2=0x7f3428141a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:54.218 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.212+0000 7f342e108640 1 --2- 172.21.15.120:0/230530991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34280b5060 0x7f3428141a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:54.218 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.212+0000 7f342e108640 1 -- 172.21.15.120:0/230530991 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3428142c50 msgr2=0x7f3428138be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:54.218 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.212+0000 7f342e108640 1 --2- 172.21.15.120:0/230530991 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3428142c50 0x7f3428138be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:54.218 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.212+0000 7f342e108640 10 monclient: _finish_hunting 0 2024-02-06T21:40:54.218 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.212+0000 7f342e108640 1 monclient: found mon.noname-a 2024-02-06T21:40:54.218 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.212+0000 7f342e108640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:40:54.218 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.212+0000 7f342e108640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:40:54.218 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.212+0000 7f342e108640 1 -- 172.21.15.120:0/230530991 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3420009a70 con 0x7f34280a9780 2024-02-06T21:40:54.218 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.214+0000 7f342e108640 1 --2- 172.21.15.120:0/230530991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34280a9780 0x7f34281db6f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f342400b4d0 tx=0x7f342400b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:54.219 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.214+0000 7f341effd640 10 client.?.objecter ms_handle_connect 0x7f34280a9780 2024-02-06T21:40:54.219 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.214+0000 7f341effd640 10 client.?.objecter resend_mon_ops 2024-02-06T21:40:54.220 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f341effd640 1 -- 172.21.15.120:0/230530991 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3424010870 con 0x7f34280a9780 2024-02-06T21:40:54.220 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f341effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:54.220 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f341effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:40:54.220 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f341effd640 10 monclient: dump: 2024-02-06T21:40:54.220 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:40:54.221 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:54.221 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:54.221 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:54.221 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:54.221 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:54.221 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:54.221 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:54.221 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:54.221 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:54.221 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f341effd640 10 monclient: _finish_auth 0 2024-02-06T21:40:54.221 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f341effd640 10 monclient: _check_auth_tickets 2024-02-06T21:40:54.221 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f341effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:40:54.221 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f341effd640 1 -- 172.21.15.120:0/230530991 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34240109d0 con 0x7f34280a9780 2024-02-06T21:40:54.221 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f341effd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:40:54.222 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f341effd640 1 -- 172.21.15.120:0/230530991 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3424010d80 con 0x7f34280a9780 2024-02-06T21:40:54.222 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f341effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:40:54.222 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f341effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:40:54.222 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f341effd640 10 monclient: dump: 2024-02-06T21:40:54.222 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:40:54.222 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:40:54.222 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:54.222 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:40:54.222 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:40:54.222 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:40:54.222 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:40:54.222 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:40:54.222 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:40:54.222 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:40:54.223 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f342c905640 4 set_mon_vals no callback set 2024-02-06T21:40:54.223 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f343036c640 5 monclient: authenticate success, global_id 4203 2024-02-06T21:40:54.223 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f343036c640 10 monclient: _renew_subs 2024-02-06T21:40:54.223 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f343036c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:54.224 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f343036c640 1 -- 172.21.15.120:0/230530991 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34281db1d0 con 0x7f34280a9780 2024-02-06T21:40:54.224 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f343036c640 10 client.4203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:40:54.224 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f343036c640 10 monclient: _renew_subs 2024-02-06T21:40:54.224 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f343036c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:54.224 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f343036c640 1 -- 172.21.15.120:0/230530991 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3428142670 con 0x7f34280a9780 2024-02-06T21:40:54.224 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.219+0000 7f343036c640 1 librados: init done 2024-02-06T21:40:54.224 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.221+0000 7f343036c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:54.224 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.221+0000 7f343036c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:54.224 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.221+0000 7f343036c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:54.224 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.221+0000 7f343036c640 1 -- 172.21.15.120:0/230530991 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34280aa1f0 con 0x7f34280a9780 2024-02-06T21:40:54.240 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.238+0000 7f341effd640 1 -- 172.21.15.120:0/230530991 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 89732+0+0 (secure 0 0 0) 0x7f3424023480 con 0x7f34280a9780 2024-02-06T21:40:54.240 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.239+0000 7f341effd640 1 --2- 172.21.15.120:0/230530991 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f340c06d160 0x7f340c06f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:40:54.240 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.239+0000 7f342d907640 1 --2- 172.21.15.120:0/230530991 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f340c06d160 0x7f340c06f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:40:54.240 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.239+0000 7f342d907640 10 monclient: get_auth_request con 0x7f340c06d160 auth_method 0 2024-02-06T21:40:54.241 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.239+0000 7f341effd640 10 client.4203.objecter ms_handle_connect 0x7f340c06d160 2024-02-06T21:40:54.241 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.239+0000 7f342d907640 1 --2- 172.21.15.120:0/230530991 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f340c06d160 0x7f340c06f620 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f34281d7690 tx=0x7f3418009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:40:54.250 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.248+0000 7f341effd640 1 -- 172.21.15.120:0/230530991 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f3424058c40 con 0x7f34280a9780 2024-02-06T21:40:54.250 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.248+0000 7f341effd640 10 client.4203.objecter ms_dispatch 0x7f3428000b90 osd_map(10..10 src has 1..10) v4 2024-02-06T21:40:54.250 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.248+0000 7f341effd640 3 client.4203.objecter handle_osd_map got epochs [10,10] > 0 2024-02-06T21:40:54.250 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.248+0000 7f341effd640 3 client.4203.objecter handle_osd_map decoding full epoch 10 2024-02-06T21:40:54.250 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.248+0000 7f341effd640 20 client.4203.objecter dump_active .. 0 homeless 2024-02-06T21:40:54.585 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.583+0000 7f341effd640 1 -- 172.21.15.120:0/230530991 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3424054ee0 con 0x7f34280a9780 2024-02-06T21:40:54.585 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.583+0000 7f341effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:40:54.585 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.583+0000 7f341effd640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:40:54.587 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:17.053 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:54.792 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.791+0000 7f343036c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:40:54.792 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.791+0000 7f343036c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:40:54.792 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.791+0000 7f343036c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:40:54.793 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.791+0000 7f343036c640 1 -- 172.21.15.120:0/230530991 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34280aa1f0 con 0x7f34280a9780 2024-02-06T21:40:54.858 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:17.297 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:54.863 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.861+0000 7f341effd640 1 -- 172.21.15.120:0/230530991 <== mon.0 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f34280aa1f0 con 0x7f34280a9780 2024-02-06T21:40:54.863 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.861+0000 7f341effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:40:54.863 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.861+0000 7f341effd640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:40:54.863 INFO:teuthology.misc.health.smithi120.stdout: 2024-02-06T21:40:54.863 INFO:teuthology.misc.health.smithi120.stdout:{"epoch":10,"fsid":"4e4791bf-0a02-4478-9517-2d2e893f4f7c","created":"2024-02-06T21:40:31.800075+0000","modified":"2024-02-06T21:40:37.354288+0000","last_up_change":"0.000000","last_in_change":"2024-02-06T21:40:37.354288+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4bd0f591-2cf1-42da-8ac1-dce3188580c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81dc801c-3768-4c2c-9adb-6c8300a341f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9289721b-c13f-4a17-be68-3698090f69af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"53534db0-2d4d-44ed-90e8-d67ea878b51a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e4205ab5-787c-44e0-9b96-273b8c9cf1ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b600bc-3afe-44bd-bd3f-690106e7e0ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1d6b9f96-1f9d-44f2-83b0-93f5c342fafb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecae00f5-8e29-47dd-90d5-e51503bc9dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-06T21:40:54.866 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 10 librados: watch_flush enter 2024-02-06T21:40:54.866 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 10 librados: watch_flush exit 2024-02-06T21:40:54.866 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 20 client.4203.objecter shutdown clearing up homeless session... 2024-02-06T21:40:54.866 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 10 client.4203.objecter successfully canceled tick 2024-02-06T21:40:54.866 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 1 -- 172.21.15.120:0/230530991 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f340c06d160 msgr2=0x7f340c06f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:54.866 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 1 --2- 172.21.15.120:0/230530991 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f340c06d160 0x7f340c06f620 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f34281d7690 tx=0x7f3418009040 comp rx=0 tx=0).stop 2024-02-06T21:40:54.867 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 10 monclient: shutdown 2024-02-06T21:40:54.867 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:40:54.867 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 1 -- 172.21.15.120:0/230530991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34280a9780 msgr2=0x7f34281db6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:40:54.867 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 1 --2- 172.21.15.120:0/230530991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34280a9780 0x7f34281db6f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f342400b4d0 tx=0x7f342400b990 comp rx=0 tx=0).stop 2024-02-06T21:40:54.867 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 1 -- 172.21.15.120:0/230530991 shutdown_connections 2024-02-06T21:40:54.867 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 1 --2- 172.21.15.120:0/230530991 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f340c06d160 0x7f340c06f620 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:54.867 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 1 --2- 172.21.15.120:0/230530991 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3428142c50 0x7f3428138be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:54.867 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 1 --2- 172.21.15.120:0/230530991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34280b5060 0x7f3428141a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:54.867 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 1 --2- 172.21.15.120:0/230530991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34280a9780 0x7f34281db6f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:40:54.867 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 1 -- 172.21.15.120:0/230530991 >> 172.21.15.120:0/230530991 conn(0x7f34280724d0 msgr2=0x7f34280728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:40:54.867 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 1 -- 172.21.15.120:0/230530991 shutdown_connections 2024-02-06T21:40:54.867 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 1 -- 172.21.15.120:0/230530991 wait complete. 2024-02-06T21:40:54.867 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:40:54.864+0000 7f343036c640 1 librados: shutdown 2024-02-06T21:40:54.876 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-02-06T21:40:54.926 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:17.372 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:55.015 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:17.454 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:55.112 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:17.559 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:55.117 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:17.575 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:55.126 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:17.593 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:40:55.574 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:18.022 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:00.383 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:22.850 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:00.387 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:22.854 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:00.738 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:23.177 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:00.742 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:23.181 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:00.755 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:23.222 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:00.759 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:23.226 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:00.838 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:23.284 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:00.842 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:23.288 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:00.861 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:23.300 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:00.865 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:23.304 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:00.877 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:23.324 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:00.881 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:23.328 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:00.884 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:23.342 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:00.890 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:23.348 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:01.307 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:23.755 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:01.312 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:23.761 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:01.682 DEBUG:teuthology.orchestra.run.smithi120:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-06T21:41:01.802 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.800+0000 7f8a3716b640 10 monclient: get_monmap_and_config 2024-02-06T21:41:01.802 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.800+0000 7f8a3716b640 10 monclient: build_initial_monmap 2024-02-06T21:41:01.802 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.800+0000 7f8a3716b640 10 monclient: monmap: 2024-02-06T21:41:01.802 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:41:01.802 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:01.802 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:41:01.801758+0000 2024-02-06T21:41:01.802 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:41:01.801758+0000 2024-02-06T21:41:01.802 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:01.802 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:01.803 INFO:teuthology.misc.health.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:01.803 INFO:teuthology.misc.health.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:01.803 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:01.803 INFO:teuthology.misc.health.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:01.803 INFO:teuthology.misc.health.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:01.803 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:01.803 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.801+0000 7f8a3716b640 1 Processor -- start 2024-02-06T21:41:01.803 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.801+0000 7f8a3716b640 1 -- start start 2024-02-06T21:41:01.803 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.801+0000 7f8a3716b640 10 monclient: init 2024-02-06T21:41:01.804 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.802+0000 7f8a3716b640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:01.804 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.802+0000 7f8a3716b640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-02-06T21:41:01.805 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.802+0000 7f8a3716b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a300b5060 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:01.805 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.802+0000 7f8a3716b640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8a3013fe70 0x7f8a30142540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:01.805 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.802+0000 7f8a3716b640 10 monclient(hunting): picked mon.noname-c con 0x7f8a3013fe70 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:01.805 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.802+0000 7f8a3716b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a300a9780 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:01.805 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.802+0000 7f8a3716b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a300a9d80 con 0x7f8a300a9780 2024-02-06T21:41:01.805 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.802+0000 7f8a3716b640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a300ab070 con 0x7f8a300b5060 2024-02-06T21:41:01.805 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.802+0000 7f8a3716b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:01.805 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.802+0000 7f8a3716b640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a300b5440 con 0x7f8a3013fe70 2024-02-06T21:41:01.805 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.802+0000 7f8a3716b640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:01.805 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.802+0000 7f8a3716b640 10 monclient(hunting): authenticate will time out at 1292.140232s 2024-02-06T21:41:01.805 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.803+0000 7f8a2ffff640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8a3013fe70 0x7f8a30142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:01.806 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.803+0000 7f8a2ffff640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8a3013fe70 0x7f8a30142540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:44914/0 (socket says 172.21.15.120:44914) 2024-02-06T21:41:01.806 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.803+0000 7f8a2ffff640 1 -- 172.21.15.120:0/1628842110 learned_addr learned my addr 172.21.15.120:0/1628842110 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:01.806 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.803+0000 7f8a2ffff640 10 monclient(hunting): get_auth_request con 0x7f8a3013fe70 auth_method 0 2024-02-06T21:41:01.806 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.803+0000 7f8a2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:01.806 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.803+0000 7f8a2ffff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:01.806 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.803+0000 7f8a2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:01.806 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.804+0000 7f8a2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:01.806 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.804+0000 7f8a2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:01.806 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.804+0000 7f8a2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:01.808 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.806+0000 7f8a2ffff640 10 monclient(hunting): handle_auth_done global_id 4198 payload 274 2024-02-06T21:41:01.808 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.806+0000 7f8a2ffff640 1 -- 172.21.15.120:0/1628842110 >> v1:172.21.15.135:6789/0 conn(0x7f8a300b5060 legacy=0x7f8a3013f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:01.808 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.806+0000 7f8a2ffff640 1 -- 172.21.15.120:0/1628842110 >> v1:172.21.15.120:6789/0 conn(0x7f8a300a9780 legacy=0x7f8a3014ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:01.808 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.806+0000 7f8a2ffff640 10 monclient: _finish_hunting 0 2024-02-06T21:41:01.808 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.806+0000 7f8a2ffff640 1 monclient: found mon.noname-c 2024-02-06T21:41:01.808 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.806+0000 7f8a2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:01.808 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.806+0000 7f8a2ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T21:41:01.808 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.806+0000 7f8a2ffff640 1 -- 172.21.15.120:0/1628842110 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a300ab350 con 0x7f8a3013fe70 2024-02-06T21:41:01.813 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.811+0000 7f8a2ffff640 1 --2- 172.21.15.120:0/1628842110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a3013fe70 0x7f8a30142540 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009870 tx=0x7f8a1c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40b3e9334457cab5 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:01.818 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.816+0000 7f8a2f7fe640 1 -- 172.21.15.120:0/1628842110 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c031870 con 0x7f8a3013fe70 2024-02-06T21:41:01.818 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.816+0000 7f8a2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:01.818 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.816+0000 7f8a2f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:01.818 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.816+0000 7f8a2f7fe640 10 monclient: dump: 2024-02-06T21:41:01.818 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:01.819 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:01.819 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:01.819 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:01.819 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:01.819 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:01.819 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:01.819 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:01.819 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:01.819 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:01.819 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.816+0000 7f8a2f7fe640 10 monclient: _finish_auth 0 2024-02-06T21:41:01.819 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.816+0000 7f8a2f7fe640 10 monclient: _check_auth_tickets 2024-02-06T21:41:01.819 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.816+0000 7f8a2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:01.819 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.816+0000 7f8a2f7fe640 1 -- 172.21.15.120:0/1628842110 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c0319d0 con 0x7f8a3013fe70 2024-02-06T21:41:01.819 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.816+0000 7f8a2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:01.819 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.816+0000 7f8a2f7fe640 1 -- 172.21.15.120:0/1628842110 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c031d60 con 0x7f8a3013fe70 2024-02-06T21:41:01.820 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.816+0000 7f8a2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:01.820 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.816+0000 7f8a2f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:01.820 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.816+0000 7f8a2f7fe640 10 monclient: dump: 2024-02-06T21:41:01.820 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:01.820 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:01.821 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:01.821 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:01.821 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:01.821 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:01.821 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:01.821 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:01.821 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:01.821 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:01.821 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.817+0000 7f8a3716b640 5 monclient: authenticate success, global_id 4198 2024-02-06T21:41:01.821 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.817+0000 7f8a3716b640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:01.821 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.817+0000 7f8a3716b640 4 set_mon_vals no callback set 2024-02-06T21:41:01.827 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.817+0000 7f8a3716b640 10 monclient: shutdown 2024-02-06T21:41:01.828 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.817+0000 7f8a3716b640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:01.828 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.817+0000 7f8a3716b640 1 -- 172.21.15.120:0/1628842110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a3013fe70 msgr2=0x7f8a30142540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:01.828 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.817+0000 7f8a3716b640 1 --2- 172.21.15.120:0/1628842110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a3013fe70 0x7f8a30142540 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009870 tx=0x7f8a1c02b910 comp rx=0 tx=0).stop 2024-02-06T21:41:01.828 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.817+0000 7f8a3716b640 1 -- 172.21.15.120:0/1628842110 shutdown_connections 2024-02-06T21:41:01.828 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.817+0000 7f8a3716b640 1 --2- 172.21.15.120:0/1628842110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a3013fe70 0x7f8a30142540 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:01.828 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.817+0000 7f8a3716b640 1 -- 172.21.15.120:0/1628842110 >> 172.21.15.120:0/1628842110 conn(0x7f8a300724d0 msgr2=0x7f8a300728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:01.828 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.817+0000 7f8a3716b640 1 -- 172.21.15.120:0/1628842110 shutdown_connections 2024-02-06T21:41:01.828 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.817+0000 7f8a3716b640 1 -- 172.21.15.120:0/1628842110 wait complete. 2024-02-06T21:41:01.829 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.818+0000 7f8a3716b640 10 monclient: build_initial_monmap 2024-02-06T21:41:01.829 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.818+0000 7f8a3716b640 10 monclient: monmap: 2024-02-06T21:41:01.829 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:41:01.829 INFO:teuthology.misc.health.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:01.829 INFO:teuthology.misc.health.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:01.829 INFO:teuthology.misc.health.smithi120.stderr:created 0.000000 2024-02-06T21:41:01.829 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:01.829 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:01.829 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:01.829 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:01.830 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:01.830 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:01.830 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.818+0000 7f8a3716b640 1 librados: starting msgr at 2024-02-06T21:41:01.830 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.818+0000 7f8a3716b640 1 librados: starting objecter 2024-02-06T21:41:01.830 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.818+0000 7f8a3716b640 1 Processor -- start 2024-02-06T21:41:01.830 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.818+0000 7f8a3716b640 1 -- start start 2024-02-06T21:41:01.830 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.818+0000 7f8a3716b640 1 librados: setting wanted keys 2024-02-06T21:41:01.830 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.818+0000 7f8a3716b640 1 librados: calling monclient init 2024-02-06T21:41:01.830 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.818+0000 7f8a3716b640 10 monclient: init 2024-02-06T21:41:01.830 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:01.830 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T21:41:01.830 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a300a9780 0x7f8a301db950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:01.830 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 10 monclient(hunting): picked mon.noname-b con 0x7f8a300a9780 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:01.830 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a300b5060 0x7f8a301dae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:01.831 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 10 monclient(hunting): picked mon.noname-c con 0x7f8a300b5060 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:01.831 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a3013fe70 0x7f8a30140be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:01.831 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 10 monclient(hunting): picked mon.noname-a con 0x7f8a3013fe70 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:01.831 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:01.831 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a301461c0 con 0x7f8a3013fe70 2024-02-06T21:41:01.831 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:01.831 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a301464a0 con 0x7f8a300a9780 2024-02-06T21:41:01.831 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:01.831 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a30146780 con 0x7f8a300b5060 2024-02-06T21:41:01.831 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:01.831 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.819+0000 7f8a3716b640 10 monclient(hunting): authenticate will time out at 1292.157124s 2024-02-06T21:41:01.831 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.820+0000 7f8a2ffff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a300b5060 0x7f8a301dae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:01.831 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.820+0000 7f8a35708640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a3013fe70 0x7f8a30140be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:01.831 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.820+0000 7f8a35708640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a3013fe70 0x7f8a30140be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:59822/0 (socket says 172.21.15.120:59822) 2024-02-06T21:41:01.831 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.820+0000 7f8a35708640 1 -- 172.21.15.120:0/1761316944 learned_addr learned my addr 172.21.15.120:0/1761316944 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:01.832 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.820+0000 7f8a35708640 10 monclient(hunting): get_auth_request con 0x7f8a3013fe70 auth_method 0 2024-02-06T21:41:01.832 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.820+0000 7f8a35708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:01.832 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.820+0000 7f8a35708640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:01.832 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.820+0000 7f8a35708640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:01.832 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.820+0000 7f8a2ffff640 10 monclient(hunting): get_auth_request con 0x7f8a300b5060 auth_method 0 2024-02-06T21:41:01.832 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.820+0000 7f8a2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:01.832 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.820+0000 7f8a2ffff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:01.832 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.820+0000 7f8a2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:01.832 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.821+0000 7f8a35708640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:01.832 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.821+0000 7f8a35708640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:01.832 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.821+0000 7f8a35708640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:01.832 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.821+0000 7f8a2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:01.832 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.821+0000 7f8a2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:01.832 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.821+0000 7f8a2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:01.833 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.821+0000 7f8a34f07640 1 --2- 172.21.15.120:0/1761316944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a300a9780 0x7f8a301db950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:01.833 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.821+0000 7f8a34f07640 10 monclient(hunting): get_auth_request con 0x7f8a300a9780 auth_method 0 2024-02-06T21:41:01.833 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.821+0000 7f8a34f07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:01.833 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.821+0000 7f8a34f07640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:01.833 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.821+0000 7f8a34f07640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:01.833 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.822+0000 7f8a34f07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:01.833 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.822+0000 7f8a34f07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:01.833 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.822+0000 7f8a34f07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:01.833 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.823+0000 7f8a35708640 10 monclient(hunting): handle_auth_done global_id 4206 payload 824 2024-02-06T21:41:01.833 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.824+0000 7f8a35708640 1 -- 172.21.15.120:0/1761316944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a300b5060 msgr2=0x7f8a301dae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:01.833 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.824+0000 7f8a35708640 1 --2- 172.21.15.120:0/1761316944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a300b5060 0x7f8a301dae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:01.834 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.824+0000 7f8a35708640 1 -- 172.21.15.120:0/1761316944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a300a9780 msgr2=0x7f8a301db950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:01.834 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.824+0000 7f8a35708640 1 --2- 172.21.15.120:0/1761316944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a300a9780 0x7f8a301db950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:01.834 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.824+0000 7f8a35708640 10 monclient: _finish_hunting 0 2024-02-06T21:41:01.834 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.824+0000 7f8a35708640 1 monclient: found mon.noname-a 2024-02-06T21:41:01.834 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.824+0000 7f8a35708640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:01.834 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.824+0000 7f8a35708640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:01.834 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.824+0000 7f8a35708640 1 -- 172.21.15.120:0/1761316944 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c0094d0 con 0x7f8a3013fe70 2024-02-06T21:41:01.834 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.826+0000 7f8a35708640 1 --2- 172.21.15.120:0/1761316944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a3013fe70 0x7f8a30140be0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800c780 tx=0x7f8a2800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:01.834 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.826+0000 7f8a2d7fa640 10 client.?.objecter ms_handle_connect 0x7f8a3013fe70 2024-02-06T21:41:01.835 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.826+0000 7f8a2d7fa640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:01.837 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a2d7fa640 1 -- 172.21.15.120:0/1761316944 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a28003d80 con 0x7f8a3013fe70 2024-02-06T21:41:01.838 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:01.838 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a2d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:01.838 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a2d7fa640 10 monclient: dump: 2024-02-06T21:41:01.838 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:01.838 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:01.838 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:01.838 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:01.838 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:01.838 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:01.838 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:01.838 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:01.838 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:01.838 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:01.838 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a2d7fa640 10 monclient: _finish_auth 0 2024-02-06T21:41:01.839 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a2d7fa640 10 monclient: _check_auth_tickets 2024-02-06T21:41:01.839 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:01.839 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a2d7fa640 1 -- 172.21.15.120:0/1761316944 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2800ce90 con 0x7f8a3013fe70 2024-02-06T21:41:01.839 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a2d7fa640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:01.839 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a2d7fa640 1 -- 172.21.15.120:0/1761316944 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a280052b0 con 0x7f8a3013fe70 2024-02-06T21:41:01.839 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a3716b640 5 monclient: authenticate success, global_id 4206 2024-02-06T21:41:01.839 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:01.839 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a2d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:01.839 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a2d7fa640 10 monclient: dump: 2024-02-06T21:41:01.839 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:01.839 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:01.839 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:01.839 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:01.839 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:01.840 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:01.840 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:01.840 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:01.840 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:01.840 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:01.840 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a2effd640 4 set_mon_vals no callback set 2024-02-06T21:41:01.840 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a3716b640 10 monclient: _renew_subs 2024-02-06T21:41:01.840 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a3716b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:01.840 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a3716b640 1 -- 172.21.15.120:0/1761316944 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a301db350 con 0x7f8a3013fe70 2024-02-06T21:41:01.840 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a3716b640 10 client.4206.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:01.840 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a3716b640 10 monclient: _renew_subs 2024-02-06T21:41:01.840 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a3716b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:01.840 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a3716b640 1 -- 172.21.15.120:0/1761316944 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a30141300 con 0x7f8a3013fe70 2024-02-06T21:41:01.841 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.830+0000 7f8a3716b640 1 librados: init done 2024-02-06T21:41:01.841 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.831+0000 7f8a3716b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:01.841 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.831+0000 7f8a3716b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:01.841 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.831+0000 7f8a3716b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:01.841 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.831+0000 7f8a3716b640 1 -- 172.21.15.120:0/1761316944 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a3006a560 con 0x7f8a3013fe70 2024-02-06T21:41:01.856 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.855+0000 7f8a2d7fa640 1 -- 172.21.15.120:0/1761316944 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 4) v1 ==== 89732+0+0 (secure 0 0 0) 0x7f8a28005410 con 0x7f8a3013fe70 2024-02-06T21:41:01.857 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.855+0000 7f8a2d7fa640 1 --2- 172.21.15.120:0/1761316944 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8a1406d160 0x7f8a1406f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:01.858 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.855+0000 7f8a34f07640 1 --2- 172.21.15.120:0/1761316944 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8a1406d160 0x7f8a1406f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:01.858 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.856+0000 7f8a34f07640 10 monclient: get_auth_request con 0x7f8a1406d160 auth_method 0 2024-02-06T21:41:01.858 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.856+0000 7f8a2d7fa640 10 client.4206.objecter ms_handle_connect 0x7f8a1406d160 2024-02-06T21:41:01.858 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:01.856+0000 7f8a34f07640 1 --2- 172.21.15.120:0/1761316944 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8a1406d160 0x7f8a1406f620 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8a24001780 tx=0x7f8a24009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:02.066 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.065+0000 7f8a2d7fa640 1 -- 172.21.15.120:0/1761316944 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f8a28058630 con 0x7f8a3013fe70 2024-02-06T21:41:02.066 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.065+0000 7f8a2d7fa640 10 client.4206.objecter ms_dispatch 0x7f8a30000b90 osd_map(10..10 src has 1..10) v4 2024-02-06T21:41:02.067 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.065+0000 7f8a2d7fa640 3 client.4206.objecter handle_osd_map got epochs [10,10] > 0 2024-02-06T21:41:02.067 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.065+0000 7f8a2d7fa640 3 client.4206.objecter handle_osd_map decoding full epoch 10 2024-02-06T21:41:02.067 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.065+0000 7f8a2d7fa640 20 client.4206.objecter dump_active .. 0 homeless 2024-02-06T21:41:02.163 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.161+0000 7f8a2d7fa640 1 -- 172.21.15.120:0/1761316944 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8a280548d0 con 0x7f8a3013fe70 2024-02-06T21:41:02.163 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.161+0000 7f8a2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:02.163 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.161+0000 7f8a2d7fa640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:02.494 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.492+0000 7f8a3716b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:02.494 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.492+0000 7f8a3716b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:02.494 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.492+0000 7f8a3716b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:02.494 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.492+0000 7f8a3716b640 1 -- 172.21.15.120:0/1761316944 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a30067c30 con 0x7f8a3013fe70 2024-02-06T21:41:02.511 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.509+0000 7f8a2d7fa640 1 -- 172.21.15.120:0/1761316944 <== mon.0 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f8a28017090 con 0x7f8a3013fe70 2024-02-06T21:41:02.511 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.509+0000 7f8a2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:02.511 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.509+0000 7f8a2d7fa640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:02.511 INFO:teuthology.misc.health.smithi120.stdout: 2024-02-06T21:41:02.512 INFO:teuthology.misc.health.smithi120.stdout:{"epoch":10,"fsid":"4e4791bf-0a02-4478-9517-2d2e893f4f7c","created":"2024-02-06T21:40:31.800075+0000","modified":"2024-02-06T21:40:37.354288+0000","last_up_change":"0.000000","last_in_change":"2024-02-06T21:40:37.354288+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4bd0f591-2cf1-42da-8ac1-dce3188580c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81dc801c-3768-4c2c-9adb-6c8300a341f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9289721b-c13f-4a17-be68-3698090f69af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"53534db0-2d4d-44ed-90e8-d67ea878b51a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e4205ab5-787c-44e0-9b96-273b8c9cf1ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b600bc-3afe-44bd-bd3f-690106e7e0ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1d6b9f96-1f9d-44f2-83b0-93f5c342fafb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecae00f5-8e29-47dd-90d5-e51503bc9dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-06T21:41:02.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.511+0000 7f8a3716b640 10 librados: watch_flush enter 2024-02-06T21:41:02.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.511+0000 7f8a3716b640 10 librados: watch_flush exit 2024-02-06T21:41:02.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.511+0000 7f8a3716b640 20 client.4206.objecter shutdown clearing up homeless session... 2024-02-06T21:41:02.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.511+0000 7f8a3716b640 10 client.4206.objecter successfully canceled tick 2024-02-06T21:41:02.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.511+0000 7f8a3716b640 1 -- 172.21.15.120:0/1761316944 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8a1406d160 msgr2=0x7f8a1406f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:02.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.511+0000 7f8a3716b640 1 --2- 172.21.15.120:0/1761316944 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8a1406d160 0x7f8a1406f620 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8a24001780 tx=0x7f8a24009040 comp rx=0 tx=0).stop 2024-02-06T21:41:02.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.511+0000 7f8a3716b640 10 monclient: shutdown 2024-02-06T21:41:02.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.511+0000 7f8a3716b640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:02.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.511+0000 7f8a3716b640 1 -- 172.21.15.120:0/1761316944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a3013fe70 msgr2=0x7f8a30140be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:02.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.511+0000 7f8a3716b640 1 --2- 172.21.15.120:0/1761316944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a3013fe70 0x7f8a30140be0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800c780 tx=0x7f8a2800cc40 comp rx=0 tx=0).stop 2024-02-06T21:41:02.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.511+0000 7f8a3716b640 1 -- 172.21.15.120:0/1761316944 shutdown_connections 2024-02-06T21:41:02.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.511+0000 7f8a3716b640 1 --2- 172.21.15.120:0/1761316944 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8a1406d160 0x7f8a1406f620 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:02.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.511+0000 7f8a3716b640 1 --2- 172.21.15.120:0/1761316944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a3013fe70 0x7f8a30140be0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:02.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.511+0000 7f8a3716b640 1 --2- 172.21.15.120:0/1761316944 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a300b5060 0x7f8a301dae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:02.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.511+0000 7f8a3716b640 1 --2- 172.21.15.120:0/1761316944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8a300a9780 0x7f8a301db950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:02.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.512+0000 7f8a3716b640 1 -- 172.21.15.120:0/1761316944 >> 172.21.15.120:0/1761316944 conn(0x7f8a300724d0 msgr2=0x7f8a3013b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:02.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.512+0000 7f8a3716b640 1 -- 172.21.15.120:0/1761316944 shutdown_connections 2024-02-06T21:41:02.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.512+0000 7f8a3716b640 1 -- 172.21.15.120:0/1761316944 wait complete. 2024-02-06T21:41:02.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:02.512+0000 7f8a3716b640 1 librados: shutdown 2024-02-06T21:41:02.523 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-02-06T21:41:04.732 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:27.199 34810== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:05.109 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:27.548 34813== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:05.329 INFO:tasks.ceph.osd.3.smithi120.stderr:==00:00:00:27.777 34832== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:05.363 INFO:tasks.ceph.osd.2.smithi120.stderr:==00:00:00:27.809 34833== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:05.374 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:27.840 34811== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:05.378 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:27.817 34812== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:05.459 INFO:tasks.ceph.osd.0.smithi120.stderr:==00:00:00:27.916 34830== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:05.751 INFO:tasks.ceph.osd.1.smithi120.stderr:==00:00:00:28.199 34831== Warning: unimplemented fcntl command: 1038 2024-02-06T21:41:09.329 DEBUG:teuthology.orchestra.run.smithi120:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-06T21:41:09.423 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.421+0000 7f8b9424d640 10 monclient: get_monmap_and_config 2024-02-06T21:41:09.423 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.421+0000 7f8b9424d640 10 monclient: build_initial_monmap 2024-02-06T21:41:09.423 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.421+0000 7f8b9424d640 10 monclient: monmap: 2024-02-06T21:41:09.423 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:41:09.424 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:09.424 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:41:09.422644+0000 2024-02-06T21:41:09.424 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:41:09.422644+0000 2024-02-06T21:41:09.424 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:09.424 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:09.424 INFO:teuthology.misc.health.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:09.424 INFO:teuthology.misc.health.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:09.424 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:09.424 INFO:teuthology.misc.health.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:09.424 INFO:teuthology.misc.health.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:09.424 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:09.424 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.421+0000 7f8b9424d640 1 Processor -- start 2024-02-06T21:41:09.424 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 1 -- start start 2024-02-06T21:41:09.424 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 10 monclient: init 2024-02-06T21:41:09.426 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:09.426 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-02-06T21:41:09.426 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8b8c0b5060 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:09.426 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8b8c13fe70 0x7f8b8c142540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:09.426 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 10 monclient(hunting): picked mon.noname-c con 0x7f8b8c13fe70 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:09.426 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b8c0a9780 0x7f8b8c142a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:09.426 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 10 monclient(hunting): picked mon.noname-b con 0x7f8b8c0a9780 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:09.427 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8c142fc0 con 0x7f8b8c0b5060 2024-02-06T21:41:09.427 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:09.427 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8c0b5440 con 0x7f8b8c13fe70 2024-02-06T21:41:09.427 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:09.427 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0a9d80 con 0x7f8b8c0a9780 2024-02-06T21:41:09.427 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:09.427 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.422+0000 7f8b9424d640 10 monclient(hunting): authenticate will time out at 1299.760485s 2024-02-06T21:41:09.427 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.423+0000 7f8b927ea640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b8c0a9780 0x7f8b8c142a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:09.427 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.423+0000 7f8b927ea640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b8c0a9780 0x7f8b8c142a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57788/0 (socket says 172.21.15.120:57788) 2024-02-06T21:41:09.427 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.423+0000 7f8b927ea640 1 -- 172.21.15.120:0/3373472603 learned_addr learned my addr 172.21.15.120:0/3373472603 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:09.427 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.423+0000 7f8b927ea640 10 monclient(hunting): get_auth_request con 0x7f8b8c0a9780 auth_method 0 2024-02-06T21:41:09.427 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.423+0000 7f8b927ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:09.427 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.423+0000 7f8b927ea640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:09.427 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.423+0000 7f8b927ea640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:09.427 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.423+0000 7f8b917e8640 1 --2- 172.21.15.120:0/3373472603 >> v2:172.21.15.135:3300/0 conn(0x7f8b8c13fe70 0x7f8b8c142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:09.428 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.424+0000 7f8b927ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:09.428 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.424+0000 7f8b927ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:09.428 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.424+0000 7f8b927ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:09.428 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.424+0000 7f8b917e8640 10 monclient(hunting): get_auth_request con 0x7f8b8c13fe70 auth_method 0 2024-02-06T21:41:09.428 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.424+0000 7f8b917e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:09.428 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.424+0000 7f8b917e8640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:09.428 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.424+0000 7f8b917e8640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:09.428 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.425+0000 7f8b917e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:09.428 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.425+0000 7f8b917e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:09.428 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.425+0000 7f8b917e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:09.429 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.426+0000 7f8b927ea640 10 monclient(hunting): handle_auth_done global_id 4193 payload 274 2024-02-06T21:41:09.429 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.426+0000 7f8b927ea640 1 -- 172.21.15.120:0/3373472603 >> v2:172.21.15.135:3300/0 conn(0x7f8b8c13fe70 msgr2=0x7f8b8c142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:09.429 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.426+0000 7f8b927ea640 1 --2- 172.21.15.120:0/3373472603 >> v2:172.21.15.135:3300/0 conn(0x7f8b8c13fe70 0x7f8b8c142540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:09.429 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.426+0000 7f8b927ea640 1 -- 172.21.15.120:0/3373472603 >> v1:172.21.15.120:6789/0 conn(0x7f8b8c0b5060 legacy=0x7f8b8c13f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:09.429 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.426+0000 7f8b927ea640 10 monclient: _finish_hunting 0 2024-02-06T21:41:09.429 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.426+0000 7f8b927ea640 1 monclient: found mon.noname-b 2024-02-06T21:41:09.429 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.426+0000 7f8b927ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:09.429 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.426+0000 7f8b927ea640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:09.429 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.426+0000 7f8b927ea640 1 -- 172.21.15.120:0/3373472603 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8c0ab070 con 0x7f8b8c0a9780 2024-02-06T21:41:09.430 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.428+0000 7f8b927ea640 1 --2- 172.21.15.120:0/3373472603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b8c0a9780 0x7f8b8c142a80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8b84009e50 tx=0x7f8b8402bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f671741ccb3a364f server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:09.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.431+0000 7f8b90fe7640 1 -- 172.21.15.120:0/3373472603 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8402ca80 con 0x7f8b8c0a9780 2024-02-06T21:41:09.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.431+0000 7f8b90fe7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:09.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.431+0000 7f8b90fe7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:09.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.431+0000 7f8b90fe7640 10 monclient: dump: 2024-02-06T21:41:09.433 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:09.433 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:09.433 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:09.434 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:09.434 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:09.434 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:09.434 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:09.434 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:09.434 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:09.434 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:09.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.431+0000 7f8b90fe7640 10 monclient: _finish_auth 0 2024-02-06T21:41:09.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.431+0000 7f8b90fe7640 10 monclient: _check_auth_tickets 2024-02-06T21:41:09.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.431+0000 7f8b90fe7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:09.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.431+0000 7f8b90fe7640 1 -- 172.21.15.120:0/3373472603 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8402cbe0 con 0x7f8b8c0a9780 2024-02-06T21:41:09.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.431+0000 7f8b90fe7640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:09.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.431+0000 7f8b90fe7640 1 -- 172.21.15.120:0/3373472603 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b840343f0 con 0x7f8b8c0a9780 2024-02-06T21:41:09.435 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.431+0000 7f8b90fe7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:09.435 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.431+0000 7f8b90fe7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:09.435 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.431+0000 7f8b90fe7640 10 monclient: dump: 2024-02-06T21:41:09.435 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:09.435 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:09.435 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:09.435 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:09.435 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:09.435 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:09.435 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:09.436 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:09.436 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:09.436 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:09.436 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.431+0000 7f8b9424d640 5 monclient: authenticate success, global_id 4193 2024-02-06T21:41:09.436 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.431+0000 7f8b9424d640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:09.436 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.432+0000 7f8b9424d640 4 set_mon_vals no callback set 2024-02-06T21:41:09.442 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.432+0000 7f8b9424d640 10 monclient: shutdown 2024-02-06T21:41:09.442 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.432+0000 7f8b9424d640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:09.442 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.432+0000 7f8b9424d640 1 -- 172.21.15.120:0/3373472603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b8c0a9780 msgr2=0x7f8b8c142a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:09.442 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.432+0000 7f8b9424d640 1 --2- 172.21.15.120:0/3373472603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b8c0a9780 0x7f8b8c142a80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8b84009e50 tx=0x7f8b8402bee0 comp rx=0 tx=0).stop 2024-02-06T21:41:09.443 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.432+0000 7f8b9424d640 1 -- 172.21.15.120:0/3373472603 shutdown_connections 2024-02-06T21:41:09.443 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.432+0000 7f8b9424d640 1 --2- 172.21.15.120:0/3373472603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b8c0a9780 0x7f8b8c142a80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:09.443 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.432+0000 7f8b9424d640 1 --2- 172.21.15.120:0/3373472603 >> v2:172.21.15.135:3300/0 conn(0x7f8b8c13fe70 0x7f8b8c142540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:09.443 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.432+0000 7f8b9424d640 1 -- 172.21.15.120:0/3373472603 >> 172.21.15.120:0/3373472603 conn(0x7f8b8c0724d0 msgr2=0x7f8b8c0728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:09.443 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.433+0000 7f8b9424d640 1 -- 172.21.15.120:0/3373472603 shutdown_connections 2024-02-06T21:41:09.443 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.433+0000 7f8b9424d640 1 -- 172.21.15.120:0/3373472603 wait complete. 2024-02-06T21:41:09.443 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.434+0000 7f8b9424d640 10 monclient: build_initial_monmap 2024-02-06T21:41:09.443 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.434+0000 7f8b9424d640 10 monclient: monmap: 2024-02-06T21:41:09.443 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:41:09.443 INFO:teuthology.misc.health.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:09.443 INFO:teuthology.misc.health.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:09.443 INFO:teuthology.misc.health.smithi120.stderr:created 0.000000 2024-02-06T21:41:09.443 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:09.443 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:09.444 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:09.444 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:09.444 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:09.444 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:09.444 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.434+0000 7f8b9424d640 1 librados: starting msgr at 2024-02-06T21:41:09.444 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.434+0000 7f8b9424d640 1 librados: starting objecter 2024-02-06T21:41:09.444 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.434+0000 7f8b9424d640 1 Processor -- start 2024-02-06T21:41:09.444 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.434+0000 7f8b9424d640 1 -- start start 2024-02-06T21:41:09.444 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.434+0000 7f8b9424d640 1 librados: setting wanted keys 2024-02-06T21:41:09.444 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.434+0000 7f8b9424d640 1 librados: calling monclient init 2024-02-06T21:41:09.444 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.434+0000 7f8b9424d640 10 monclient: init 2024-02-06T21:41:09.445 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:09.445 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T21:41:09.445 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b8c0a9780 0x7f8b8c06e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:09.445 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 10 monclient(hunting): picked mon.noname-c con 0x7f8b8c0a9780 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:09.445 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b8c0b5060 0x7f8b8c071630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:09.445 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 10 monclient(hunting): picked mon.noname-b con 0x7f8b8c0b5060 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:09.445 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b8c13fe70 0x7f8b8c071b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:09.445 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 10 monclient(hunting): picked mon.noname-a con 0x7f8b8c13fe70 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:09.445 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:09.445 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c143b60 con 0x7f8b8c13fe70 2024-02-06T21:41:09.445 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:09.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c06f6a0 con 0x7f8b8c0b5060 2024-02-06T21:41:09.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:09.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0720b0 con 0x7f8b8c0a9780 2024-02-06T21:41:09.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:09.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.435+0000 7f8b9424d640 10 monclient(hunting): authenticate will time out at 1299.773440s 2024-02-06T21:41:09.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b91fe9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b8c0a9780 0x7f8b8c06e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:09.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b927ea640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b8c13fe70 0x7f8b8c071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:09.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b917e8640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b8c0b5060 0x7f8b8c071630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:09.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b91fe9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b8c0a9780 0x7f8b8c06e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57790/0 (socket says 172.21.15.120:57790) 2024-02-06T21:41:09.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b91fe9640 1 -- 172.21.15.120:0/131189431 learned_addr learned my addr 172.21.15.120:0/131189431 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:09.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b91fe9640 10 monclient(hunting): get_auth_request con 0x7f8b8c0a9780 auth_method 0 2024-02-06T21:41:09.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b91fe9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:09.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b91fe9640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:09.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b91fe9640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:09.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b927ea640 10 monclient(hunting): get_auth_request con 0x7f8b8c13fe70 auth_method 0 2024-02-06T21:41:09.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b927ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:09.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b927ea640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:09.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b927ea640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:09.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b917e8640 10 monclient(hunting): get_auth_request con 0x7f8b8c0b5060 auth_method 0 2024-02-06T21:41:09.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b917e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:09.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b917e8640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:09.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.436+0000 7f8b917e8640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:09.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.437+0000 7f8b91fe9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:09.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.437+0000 7f8b91fe9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:09.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.437+0000 7f8b91fe9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:09.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.437+0000 7f8b927ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:09.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.437+0000 7f8b927ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:09.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.437+0000 7f8b927ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:09.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.437+0000 7f8b917e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:09.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.437+0000 7f8b917e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:09.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.437+0000 7f8b917e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:09.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.439+0000 7f8b91fe9640 10 monclient(hunting): handle_auth_done global_id 4196 payload 824 2024-02-06T21:41:09.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.439+0000 7f8b91fe9640 1 -- 172.21.15.120:0/131189431 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b8c0b5060 msgr2=0x7f8b8c071630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:09.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.439+0000 7f8b91fe9640 1 --2- 172.21.15.120:0/131189431 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b8c0b5060 0x7f8b8c071630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:09.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.439+0000 7f8b91fe9640 1 -- 172.21.15.120:0/131189431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b8c13fe70 msgr2=0x7f8b8c071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:09.449 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.440+0000 7f8b91fe9640 1 --2- 172.21.15.120:0/131189431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b8c13fe70 0x7f8b8c071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:09.449 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.440+0000 7f8b91fe9640 10 monclient: _finish_hunting 0 2024-02-06T21:41:09.449 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.440+0000 7f8b91fe9640 1 monclient: found mon.noname-c 2024-02-06T21:41:09.449 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.440+0000 7f8b91fe9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:09.449 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.440+0000 7f8b91fe9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:41:09.449 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.440+0000 7f8b91fe9640 1 -- 172.21.15.120:0/131189431 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b84009b30 con 0x7f8b8c0a9780 2024-02-06T21:41:09.452 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.442+0000 7f8b827fc640 10 client.?.objecter ms_handle_connect 0x7f8b8c0a9780 2024-02-06T21:41:09.452 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.442+0000 7f8b827fc640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:09.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.442+0000 7f8b91fe9640 1 --2- 172.21.15.120:0/131189431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b8c0a9780 0x7f8b8c06e630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8b8800c430 tx=0x7f8b8800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:09.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b827fc640 1 -- 172.21.15.120:0/131189431 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b880052d0 con 0x7f8b8c0a9780 2024-02-06T21:41:09.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:09.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:09.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b827fc640 10 monclient: dump: 2024-02-06T21:41:09.453 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:09.453 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:09.453 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:09.453 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:09.453 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:09.453 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:09.453 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:09.453 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:09.454 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:09.454 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:09.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b827fc640 10 monclient: _finish_auth 0 2024-02-06T21:41:09.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b827fc640 10 monclient: _check_auth_tickets 2024-02-06T21:41:09.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:09.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b9424d640 5 monclient: authenticate success, global_id 4196 2024-02-06T21:41:09.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b9424d640 10 monclient: _renew_subs 2024-02-06T21:41:09.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b9424d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:09.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b9424d640 1 -- 172.21.15.120:0/131189431 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b8c06f910 con 0x7f8b8c0a9780 2024-02-06T21:41:09.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b9424d640 10 client.4196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:09.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b9424d640 10 monclient: _renew_subs 2024-02-06T21:41:09.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b9424d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:09.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b9424d640 1 -- 172.21.15.120:0/131189431 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8c06fed0 con 0x7f8b8c0a9780 2024-02-06T21:41:09.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.445+0000 7f8b9424d640 1 librados: init done 2024-02-06T21:41:09.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.446+0000 7f8b827fc640 1 -- 172.21.15.120:0/131189431 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b88005430 con 0x7f8b8c0a9780 2024-02-06T21:41:09.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.446+0000 7f8b827fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:09.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.446+0000 7f8b827fc640 1 -- 172.21.15.120:0/131189431 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b880052d0 con 0x7f8b8c0a9780 2024-02-06T21:41:09.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.446+0000 7f8b827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:09.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.446+0000 7f8b827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:09.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.446+0000 7f8b827fc640 10 monclient: dump: 2024-02-06T21:41:09.455 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:09.455 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:09.455 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:09.455 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:09.455 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:09.455 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:09.455 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:09.456 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:09.456 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:09.456 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:09.456 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.446+0000 7f8b83fff640 4 set_mon_vals no callback set 2024-02-06T21:41:09.456 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.446+0000 7f8b9424d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:09.456 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.446+0000 7f8b9424d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:09.456 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.446+0000 7f8b9424d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:09.456 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.446+0000 7f8b9424d640 1 -- 172.21.15.120:0/131189431 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b8c06a560 con 0x7f8b8c0a9780 2024-02-06T21:41:09.477 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.475+0000 7f8b827fc640 1 -- 172.21.15.120:0/131189431 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 89732+0+0 (secure 0 0 0) 0x7f8b88007020 con 0x7f8b8c0a9780 2024-02-06T21:41:09.477 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.476+0000 7f8b827fc640 1 --2- 172.21.15.120:0/131189431 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8b7006d170 0x7f8b7006f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:09.478 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.476+0000 7f8b917e8640 1 --2- 172.21.15.120:0/131189431 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8b7006d170 0x7f8b7006f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:09.478 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.476+0000 7f8b917e8640 10 monclient: get_auth_request con 0x7f8b7006d170 auth_method 0 2024-02-06T21:41:09.478 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.477+0000 7f8b827fc640 10 client.4196.objecter ms_handle_connect 0x7f8b7006d170 2024-02-06T21:41:09.478 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.477+0000 7f8b917e8640 1 --2- 172.21.15.120:0/131189431 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8b7006d170 0x7f8b7006f630 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c1e01e0 tx=0x7f8b7c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:09.482 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.480+0000 7f8b827fc640 1 -- 172.21.15.120:0/131189431 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f8b88057f00 con 0x7f8b8c0a9780 2024-02-06T21:41:09.482 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.480+0000 7f8b827fc640 10 client.4196.objecter ms_dispatch 0x7f8b8c000b90 osd_map(10..10 src has 1..10) v4 2024-02-06T21:41:09.482 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.480+0000 7f8b827fc640 3 client.4196.objecter handle_osd_map got epochs [10,10] > 0 2024-02-06T21:41:09.482 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.480+0000 7f8b827fc640 3 client.4196.objecter handle_osd_map decoding full epoch 10 2024-02-06T21:41:09.482 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.481+0000 7f8b827fc640 20 client.4196.objecter dump_active .. 0 homeless 2024-02-06T21:41:09.731 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.729+0000 7f8b827fc640 1 -- 172.21.15.120:0/131189431 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8b880541a0 con 0x7f8b8c0a9780 2024-02-06T21:41:09.731 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.729+0000 7f8b827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:09.731 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.729+0000 7f8b827fc640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:09.971 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.969+0000 7f8b9424d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:09.971 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.969+0000 7f8b9424d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:09.971 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.969+0000 7f8b9424d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:09.971 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.969+0000 7f8b9424d640 1 -- 172.21.15.120:0/131189431 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b8c067c30 con 0x7f8b8c0a9780 2024-02-06T21:41:09.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.997+0000 7f8b827fc640 1 -- 172.21.15.120:0/131189431 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f8b88020090 con 0x7f8b8c0a9780 2024-02-06T21:41:09.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.997+0000 7f8b827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:09.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:09.997+0000 7f8b827fc640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:09.999 INFO:teuthology.misc.health.smithi120.stdout: 2024-02-06T21:41:09.999 INFO:teuthology.misc.health.smithi120.stdout:{"epoch":10,"fsid":"4e4791bf-0a02-4478-9517-2d2e893f4f7c","created":"2024-02-06T21:40:31.800075+0000","modified":"2024-02-06T21:40:37.354288+0000","last_up_change":"0.000000","last_in_change":"2024-02-06T21:40:37.354288+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4bd0f591-2cf1-42da-8ac1-dce3188580c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81dc801c-3768-4c2c-9adb-6c8300a341f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9289721b-c13f-4a17-be68-3698090f69af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"53534db0-2d4d-44ed-90e8-d67ea878b51a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e4205ab5-787c-44e0-9b96-273b8c9cf1ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b600bc-3afe-44bd-bd3f-690106e7e0ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1d6b9f96-1f9d-44f2-83b0-93f5c342fafb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecae00f5-8e29-47dd-90d5-e51503bc9dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-06T21:41:10.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 10 librados: watch_flush enter 2024-02-06T21:41:10.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 10 librados: watch_flush exit 2024-02-06T21:41:10.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 20 client.4196.objecter shutdown clearing up homeless session... 2024-02-06T21:41:10.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 10 client.4196.objecter successfully canceled tick 2024-02-06T21:41:10.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 1 -- 172.21.15.120:0/131189431 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8b7006d170 msgr2=0x7f8b7006f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:10.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 1 --2- 172.21.15.120:0/131189431 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8b7006d170 0x7f8b7006f630 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c1e01e0 tx=0x7f8b7c009190 comp rx=0 tx=0).stop 2024-02-06T21:41:10.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 10 monclient: shutdown 2024-02-06T21:41:10.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:10.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 1 -- 172.21.15.120:0/131189431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b8c0a9780 msgr2=0x7f8b8c06e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:10.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 1 --2- 172.21.15.120:0/131189431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b8c0a9780 0x7f8b8c06e630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8b8800c430 tx=0x7f8b8800c8f0 comp rx=0 tx=0).stop 2024-02-06T21:41:10.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 1 -- 172.21.15.120:0/131189431 shutdown_connections 2024-02-06T21:41:10.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 1 --2- 172.21.15.120:0/131189431 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8b7006d170 0x7f8b7006f630 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:10.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 1 --2- 172.21.15.120:0/131189431 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b8c0b5060 0x7f8b8c071630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:10.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 1 --2- 172.21.15.120:0/131189431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b8c13fe70 0x7f8b8c071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:10.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 1 --2- 172.21.15.120:0/131189431 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b8c0a9780 0x7f8b8c06e630 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:10.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.000+0000 7f8b9424d640 1 -- 172.21.15.120:0/131189431 >> 172.21.15.120:0/131189431 conn(0x7f8b8c0724d0 msgr2=0x7f8b8c0b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:10.004 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.001+0000 7f8b9424d640 1 -- 172.21.15.120:0/131189431 shutdown_connections 2024-02-06T21:41:10.004 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.001+0000 7f8b9424d640 1 -- 172.21.15.120:0/131189431 wait complete. 2024-02-06T21:41:10.004 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:10.001+0000 7f8b9424d640 1 librados: shutdown 2024-02-06T21:41:10.017 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-02-06T21:41:13.732 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:41:13.726+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-02-06T21:41:13.760 INFO:tasks.ceph.osd.7.smithi135.stderr:2024-02-06T21:41:13.698+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-02-06T21:41:13.946 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:41:13.940+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-02-06T21:41:14.157 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:14.152+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-02-06T21:41:14.161 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:41:14.157+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-02-06T21:41:14.258 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:41:14.254+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-02-06T21:41:14.268 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:41:14.263+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-02-06T21:41:15.043 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:15.039+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-02-06T21:41:16.824 DEBUG:teuthology.orchestra.run.smithi120:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-06T21:41:16.921 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.919+0000 7fd1e6015640 10 monclient: get_monmap_and_config 2024-02-06T21:41:16.921 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.919+0000 7fd1e6015640 10 monclient: build_initial_monmap 2024-02-06T21:41:16.921 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.919+0000 7fd1e6015640 10 monclient: monmap: 2024-02-06T21:41:16.921 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:41:16.921 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:16.921 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:41:16.920584+0000 2024-02-06T21:41:16.921 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:41:16.920584+0000 2024-02-06T21:41:16.921 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:16.921 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:16.921 INFO:teuthology.misc.health.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:16.921 INFO:teuthology.misc.health.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:16.922 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:16.922 INFO:teuthology.misc.health.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:16.922 INFO:teuthology.misc.health.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:16.922 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:16.923 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.920+0000 7fd1e6015640 1 Processor -- start 2024-02-06T21:41:16.923 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.920+0000 7fd1e6015640 1 -- start start 2024-02-06T21:41:16.923 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.920+0000 7fd1e6015640 10 monclient: init 2024-02-06T21:41:16.923 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.920+0000 7fd1e6015640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:16.923 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.920+0000 7fd1e6015640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-02-06T21:41:16.924 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.920+0000 7fd1e6015640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e00b5060 0x7fd1e013f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:16.924 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.920+0000 7fd1e6015640 10 monclient(hunting): picked mon.noname-b con 0x7fd1e00b5060 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:16.924 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.920+0000 7fd1e6015640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1e00a9780 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:16.924 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.920+0000 7fd1e6015640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1e0142c50 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:16.924 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.920+0000 7fd1e6015640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e00a9d80 con 0x7fd1e0142c50 2024-02-06T21:41:16.924 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.920+0000 7fd1e6015640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e00ab070 con 0x7fd1e00a9780 2024-02-06T21:41:16.924 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.920+0000 7fd1e6015640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:16.924 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.920+0000 7fd1e6015640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e00b5440 con 0x7fd1e00b5060 2024-02-06T21:41:16.924 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.920+0000 7fd1e6015640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:16.924 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.920+0000 7fd1e6015640 10 monclient(hunting): authenticate will time out at 1307.258507s 2024-02-06T21:41:16.924 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.921+0000 7fd1deffd640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fd1e00a9780 0x7fd1e0142540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.120:52328/0 (socket says 172.21.15.120:52328) 2024-02-06T21:41:16.924 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.921+0000 7fd1deffd640 1 -- 172.21.15.120:0/769051566 learned_addr learned my addr 172.21.15.120:0/769051566 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:41:16.924 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.921+0000 7fd1df7fe640 1 --2- 172.21.15.120:0/769051566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e00b5060 0x7fd1e013f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:16.925 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.922+0000 7fd1df7fe640 10 monclient(hunting): get_auth_request con 0x7fd1e00b5060 auth_method 0 2024-02-06T21:41:16.925 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.922+0000 7fd1df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:16.925 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.922+0000 7fd1df7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:16.925 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.922+0000 7fd1df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:16.925 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.922+0000 7fd1df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:16.925 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.922+0000 7fd1df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:16.925 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.922+0000 7fd1df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:16.926 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.925+0000 7fd1df7fe640 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2024-02-06T21:41:16.926 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.925+0000 7fd1df7fe640 1 -- 172.21.15.120:0/769051566 >> v1:172.21.15.135:6789/0 conn(0x7fd1e00a9780 legacy=0x7fd1e0142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:16.926 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.925+0000 7fd1df7fe640 1 -- 172.21.15.120:0/769051566 >> v1:172.21.15.120:6789/0 conn(0x7fd1e0142c50 legacy=0x7fd1e014ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:16.926 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.925+0000 7fd1df7fe640 10 monclient: _finish_hunting 0 2024-02-06T21:41:16.926 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.925+0000 7fd1df7fe640 1 monclient: found mon.noname-b 2024-02-06T21:41:16.927 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.925+0000 7fd1df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:16.927 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.925+0000 7fd1df7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:16.927 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.925+0000 7fd1df7fe640 1 -- 172.21.15.120:0/769051566 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e00ab350 con 0x7fd1e00b5060 2024-02-06T21:41:16.930 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.928+0000 7fd1df7fe640 1 --2- 172.21.15.120:0/769051566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e00b5060 0x7fd1e013f760 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd1d00097d0 tx=0x7fd1d002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e34d32aab541a013 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:16.934 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1de7fc640 1 -- 172.21.15.120:0/769051566 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0031780 con 0x7fd1e00b5060 2024-02-06T21:41:16.934 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:16.934 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1de7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:16.935 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1de7fc640 10 monclient: dump: 2024-02-06T21:41:16.935 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:16.935 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:16.935 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:16.935 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:16.935 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:16.935 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:16.935 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:16.935 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:16.935 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:16.935 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:16.935 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1de7fc640 10 monclient: _finish_auth 0 2024-02-06T21:41:16.935 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1de7fc640 10 monclient: _check_auth_tickets 2024-02-06T21:41:16.935 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:16.935 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1de7fc640 1 -- 172.21.15.120:0/769051566 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d00318e0 con 0x7fd1e00b5060 2024-02-06T21:41:16.936 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1de7fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:16.936 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1de7fc640 1 -- 172.21.15.120:0/769051566 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0031c70 con 0x7fd1e00b5060 2024-02-06T21:41:16.936 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:16.936 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:16.936 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1de7fc640 10 monclient: dump: 2024-02-06T21:41:16.936 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:16.936 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:16.936 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:16.936 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:16.936 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:16.936 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:16.936 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:16.936 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:16.936 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:16.937 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:16.937 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1e6015640 5 monclient: authenticate success, global_id 4223 2024-02-06T21:41:16.937 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1e6015640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:16.937 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.932+0000 7fd1e6015640 4 set_mon_vals no callback set 2024-02-06T21:41:16.937 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.933+0000 7fd1e6015640 10 monclient: shutdown 2024-02-06T21:41:16.937 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.933+0000 7fd1e6015640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:16.937 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.933+0000 7fd1e6015640 1 -- 172.21.15.120:0/769051566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e00b5060 msgr2=0x7fd1e013f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:16.937 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.933+0000 7fd1e6015640 1 --2- 172.21.15.120:0/769051566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e00b5060 0x7fd1e013f760 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd1d00097d0 tx=0x7fd1d002b820 comp rx=0 tx=0).stop 2024-02-06T21:41:16.939 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.934+0000 7fd1e6015640 1 -- 172.21.15.120:0/769051566 shutdown_connections 2024-02-06T21:41:16.940 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.934+0000 7fd1e6015640 1 --2- 172.21.15.120:0/769051566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e00b5060 0x7fd1e013f760 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:16.940 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.934+0000 7fd1e6015640 1 -- 172.21.15.120:0/769051566 >> 172.21.15.120:0/769051566 conn(0x7fd1e00724d0 msgr2=0x7fd1e00728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:16.940 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.934+0000 7fd1e6015640 1 -- 172.21.15.120:0/769051566 shutdown_connections 2024-02-06T21:41:16.940 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.934+0000 7fd1e6015640 1 -- 172.21.15.120:0/769051566 wait complete. 2024-02-06T21:41:16.940 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.935+0000 7fd1e6015640 10 monclient: build_initial_monmap 2024-02-06T21:41:16.940 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.935+0000 7fd1e6015640 10 monclient: monmap: 2024-02-06T21:41:16.940 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:41:16.940 INFO:teuthology.misc.health.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:16.940 INFO:teuthology.misc.health.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:16.940 INFO:teuthology.misc.health.smithi120.stderr:created 0.000000 2024-02-06T21:41:16.940 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:16.941 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:16.941 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:16.941 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:16.941 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:16.941 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:16.941 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.935+0000 7fd1e6015640 1 librados: starting msgr at 2024-02-06T21:41:16.941 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.935+0000 7fd1e6015640 1 librados: starting objecter 2024-02-06T21:41:16.941 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.935+0000 7fd1e6015640 1 Processor -- start 2024-02-06T21:41:16.941 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.936+0000 7fd1e6015640 1 -- start start 2024-02-06T21:41:16.941 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.936+0000 7fd1e6015640 1 librados: setting wanted keys 2024-02-06T21:41:16.941 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.936+0000 7fd1e6015640 1 librados: calling monclient init 2024-02-06T21:41:16.941 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.936+0000 7fd1e6015640 10 monclient: init 2024-02-06T21:41:16.941 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.936+0000 7fd1e6015640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:16.941 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.936+0000 7fd1e6015640 10 monclient: _add_conns ranks=[2,0,1] 2024-02-06T21:41:16.941 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.936+0000 7fd1e6015640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e00a9780 0x7fd1e00b74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:16.942 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.936+0000 7fd1e6015640 10 monclient(hunting): picked mon.noname-c con 0x7fd1e00a9780 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:16.942 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.936+0000 7fd1e6015640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1e00b5060 0x7fd1e00b6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:16.942 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.937+0000 7fd1e6015640 10 monclient(hunting): picked mon.noname-a con 0x7fd1e00b5060 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:16.942 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.937+0000 7fd1e6015640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1e0142c50 0x7fd1e01d7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:16.942 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.937+0000 7fd1e6015640 10 monclient(hunting): picked mon.noname-b con 0x7fd1e0142c50 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:16.942 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.937+0000 7fd1e6015640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:16.942 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.937+0000 7fd1e6015640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e00b8b10 con 0x7fd1e00b5060 2024-02-06T21:41:16.942 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.937+0000 7fd1e6015640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:16.942 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.937+0000 7fd1e6015640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e01d77c0 con 0x7fd1e0142c50 2024-02-06T21:41:16.942 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.937+0000 7fd1e6015640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:16.942 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.937+0000 7fd1e6015640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e00b6c60 con 0x7fd1e00a9780 2024-02-06T21:41:16.942 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.937+0000 7fd1e6015640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:16.942 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.937+0000 7fd1e6015640 10 monclient(hunting): authenticate will time out at 1307.274590s 2024-02-06T21:41:16.942 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.937+0000 7fd1dffff640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1e0142c50 0x7fd1e01d7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:16.942 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.937+0000 7fd1df7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e00a9780 0x7fd1e00b74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:16.943 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1dffff640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1e0142c50 0x7fd1e01d7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:42086/0 (socket says 172.21.15.120:42086) 2024-02-06T21:41:16.943 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1dffff640 1 -- 172.21.15.120:0/1719735437 learned_addr learned my addr 172.21.15.120:0/1719735437 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:16.943 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1dffff640 10 monclient(hunting): get_auth_request con 0x7fd1e0142c50 auth_method 0 2024-02-06T21:41:16.943 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:16.943 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1dffff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:16.943 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1dffff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:16.943 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1df7fe640 10 monclient(hunting): get_auth_request con 0x7fd1e00a9780 auth_method 0 2024-02-06T21:41:16.943 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:16.943 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1df7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:16.943 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:16.943 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:16.944 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:16.944 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:16.944 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:16.944 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:16.944 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.938+0000 7fd1df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:16.945 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.941+0000 7fd1dffff640 10 monclient(hunting): handle_auth_done global_id 4237 payload 824 2024-02-06T21:41:16.945 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.941+0000 7fd1dffff640 1 -- 172.21.15.120:0/1719735437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e00a9780 msgr2=0x7fd1e00b74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:16.945 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.941+0000 7fd1dffff640 1 --2- 172.21.15.120:0/1719735437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e00a9780 0x7fd1e00b74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:16.945 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.941+0000 7fd1dffff640 1 -- 172.21.15.120:0/1719735437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1e00b5060 msgr2=0x7fd1e00b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:16.945 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.941+0000 7fd1dffff640 1 --2- 172.21.15.120:0/1719735437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1e00b5060 0x7fd1e00b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:16.945 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.941+0000 7fd1dffff640 10 monclient: _finish_hunting 0 2024-02-06T21:41:16.945 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.941+0000 7fd1dffff640 1 monclient: found mon.noname-b 2024-02-06T21:41:16.945 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.941+0000 7fd1dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:16.945 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.941+0000 7fd1dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:16.945 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.941+0000 7fd1dffff640 1 -- 172.21.15.120:0/1719735437 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d0009400 con 0x7fd1e0142c50 2024-02-06T21:41:16.951 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.948+0000 7fd1dffff640 1 --2- 172.21.15.120:0/1719735437 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1e0142c50 0x7fd1e01d7ce0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd1d800b990 tx=0x7fd1d800be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:16.951 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.948+0000 7fd1c7fff640 10 client.?.objecter ms_handle_connect 0x7fd1e0142c50 2024-02-06T21:41:16.951 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.948+0000 7fd1c7fff640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:16.955 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1c7fff640 1 -- 172.21.15.120:0/1719735437 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d800cba0 con 0x7fd1e0142c50 2024-02-06T21:41:16.955 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:16.955 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:16.955 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1c7fff640 10 monclient: dump: 2024-02-06T21:41:16.955 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:16.956 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:16.956 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:16.956 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:16.956 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:16.956 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:16.956 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:16.956 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:16.956 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:16.956 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:16.956 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1c7fff640 10 monclient: _finish_auth 0 2024-02-06T21:41:16.956 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1c7fff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:16.956 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:16.957 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1c7fff640 1 -- 172.21.15.120:0/1719735437 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d800cd00 con 0x7fd1e0142c50 2024-02-06T21:41:16.957 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1c7fff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:16.957 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1c7fff640 1 -- 172.21.15.120:0/1719735437 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8014680 con 0x7fd1e0142c50 2024-02-06T21:41:16.957 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:16.957 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:16.957 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1c7fff640 10 monclient: dump: 2024-02-06T21:41:16.957 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:16.957 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:16.957 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:16.957 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:16.957 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:16.957 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:16.957 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:16.958 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:16.958 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:16.958 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:16.958 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1e6015640 5 monclient: authenticate success, global_id 4237 2024-02-06T21:41:16.958 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1e6015640 10 monclient: _renew_subs 2024-02-06T21:41:16.958 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1e6015640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:16.958 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1e6015640 1 -- 172.21.15.120:0/1719735437 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e0144a70 con 0x7fd1e0142c50 2024-02-06T21:41:16.958 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1e6015640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:16.958 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1e6015640 10 monclient: _renew_subs 2024-02-06T21:41:16.958 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1ddffb640 4 set_mon_vals no callback set 2024-02-06T21:41:16.958 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1e6015640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:16.958 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1e6015640 1 -- 172.21.15.120:0/1719735437 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e01d8400 con 0x7fd1e0142c50 2024-02-06T21:41:16.958 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.953+0000 7fd1e6015640 1 librados: init done 2024-02-06T21:41:16.959 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.955+0000 7fd1e6015640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:16.959 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.955+0000 7fd1e6015640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:16.959 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.955+0000 7fd1e6015640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:16.959 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.955+0000 7fd1e6015640 1 -- 172.21.15.120:0/1719735437 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e00aa1f0 con 0x7fd1e0142c50 2024-02-06T21:41:16.985 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.983+0000 7fd1c7fff640 1 -- 172.21.15.120:0/1719735437 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 4) v1 ==== 89732+0+0 (secure 0 0 0) 0x7fd1d801d3b0 con 0x7fd1e0142c50 2024-02-06T21:41:16.985 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.984+0000 7fd1c7fff640 1 --2- 172.21.15.120:0/1719735437 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd1cc06d110 0x7fd1cc06f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:16.985 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.984+0000 7fd1df7fe640 1 --2- 172.21.15.120:0/1719735437 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd1cc06d110 0x7fd1cc06f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:16.985 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.984+0000 7fd1df7fe640 10 monclient: get_auth_request con 0x7fd1cc06d110 auth_method 0 2024-02-06T21:41:16.986 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.984+0000 7fd1c7fff640 10 client.4237.objecter ms_handle_connect 0x7fd1cc06d110 2024-02-06T21:41:16.986 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:16.984+0000 7fd1df7fe640 1 --2- 172.21.15.120:0/1719735437 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd1cc06d110 0x7fd1cc06f5d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0036000 tx=0x7fd1d003d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:17.252 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.250+0000 7fd1c7fff640 1 -- 172.21.15.120:0/1719735437 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2215+0+0 (secure 0 0 0) 0x7fd1d8058c10 con 0x7fd1e0142c50 2024-02-06T21:41:17.253 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.250+0000 7fd1c7fff640 10 client.4237.objecter ms_dispatch 0x7fd1e0000b90 osd_map(12..12 src has 1..12) v4 2024-02-06T21:41:17.253 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.250+0000 7fd1c7fff640 3 client.4237.objecter handle_osd_map got epochs [12,12] > 0 2024-02-06T21:41:17.253 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.250+0000 7fd1c7fff640 3 client.4237.objecter handle_osd_map decoding full epoch 12 2024-02-06T21:41:17.253 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.251+0000 7fd1c7fff640 20 client.4237.objecter dump_active .. 0 homeless 2024-02-06T21:41:17.253 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.251+0000 7fd1c7fff640 1 -- 172.21.15.120:0/1719735437 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd1d8054eb0 con 0x7fd1e0142c50 2024-02-06T21:41:17.253 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.251+0000 7fd1c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:17.253 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.251+0000 7fd1c7fff640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:17.347 INFO:tasks.ceph.osd.7.smithi135.stderr:2024-02-06T21:41:17.342+0000 c48f640 -1 osd.7 0 waiting for initial osdmap 2024-02-06T21:41:17.406 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:41:17.400+0000 ac79640 -1 osd.0 0 waiting for initial osdmap 2024-02-06T21:41:17.483 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.481+0000 7fd1e6015640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:17.483 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.481+0000 7fd1e6015640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:17.483 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.481+0000 7fd1e6015640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:17.483 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.481+0000 7fd1e6015640 1 -- 172.21.15.120:0/1719735437 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1e00aa1f0 con 0x7fd1e0142c50 2024-02-06T21:41:17.511 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.510+0000 7fd1c7fff640 1 -- 172.21.15.120:0/1719735437 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fd1e00aa1f0 con 0x7fd1e0142c50 2024-02-06T21:41:17.511 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.510+0000 7fd1c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:17.511 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.510+0000 7fd1c7fff640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:17.511 INFO:teuthology.misc.health.smithi120.stdout: 2024-02-06T21:41:17.512 INFO:teuthology.misc.health.smithi120.stdout:{"epoch":12,"fsid":"4e4791bf-0a02-4478-9517-2d2e893f4f7c","created":"2024-02-06T21:40:31.800075+0000","modified":"2024-02-06T21:41:16.790396+0000","last_up_change":"0.000000","last_in_change":"2024-02-06T21:40:37.354288+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4bd0f591-2cf1-42da-8ac1-dce3188580c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81dc801c-3768-4c2c-9adb-6c8300a341f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9289721b-c13f-4a17-be68-3698090f69af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"53534db0-2d4d-44ed-90e8-d67ea878b51a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e4205ab5-787c-44e0-9b96-273b8c9cf1ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b600bc-3afe-44bd-bd3f-690106e7e0ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1d6b9f96-1f9d-44f2-83b0-93f5c342fafb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecae00f5-8e29-47dd-90d5-e51503bc9dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-06T21:41:17.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.511+0000 7fd1e6015640 10 librados: watch_flush enter 2024-02-06T21:41:17.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.511+0000 7fd1e6015640 10 librados: watch_flush exit 2024-02-06T21:41:17.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 20 client.4237.objecter shutdown clearing up homeless session... 2024-02-06T21:41:17.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 10 client.4237.objecter successfully canceled tick 2024-02-06T21:41:17.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 1 -- 172.21.15.120:0/1719735437 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd1cc06d110 msgr2=0x7fd1cc06f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:17.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 1 --2- 172.21.15.120:0/1719735437 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd1cc06d110 0x7fd1cc06f5d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0036000 tx=0x7fd1d003d040 comp rx=0 tx=0).stop 2024-02-06T21:41:17.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 10 monclient: shutdown 2024-02-06T21:41:17.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:17.514 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 1 -- 172.21.15.120:0/1719735437 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1e0142c50 msgr2=0x7fd1e01d7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:17.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 1 --2- 172.21.15.120:0/1719735437 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1e0142c50 0x7fd1e01d7ce0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd1d800b990 tx=0x7fd1d800be50 comp rx=0 tx=0).stop 2024-02-06T21:41:17.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 1 -- 172.21.15.120:0/1719735437 shutdown_connections 2024-02-06T21:41:17.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 1 --2- 172.21.15.120:0/1719735437 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd1cc06d110 0x7fd1cc06f5d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:17.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 1 --2- 172.21.15.120:0/1719735437 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1e0142c50 0x7fd1e01d7ce0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:17.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 1 --2- 172.21.15.120:0/1719735437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1e00b5060 0x7fd1e00b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:17.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 1 --2- 172.21.15.120:0/1719735437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e00a9780 0x7fd1e00b74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:17.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 1 -- 172.21.15.120:0/1719735437 >> 172.21.15.120:0/1719735437 conn(0x7fd1e00724d0 msgr2=0x7fd1e00b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:17.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 1 -- 172.21.15.120:0/1719735437 shutdown_connections 2024-02-06T21:41:17.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 1 -- 172.21.15.120:0/1719735437 wait complete. 2024-02-06T21:41:17.515 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:17.512+0000 7fd1e6015640 1 librados: shutdown 2024-02-06T21:41:17.523 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-02-06T21:41:17.806 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:41:17.785+0000 10255640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-06T21:41:17.842 INFO:tasks.ceph.osd.7.smithi135.stderr:2024-02-06T21:41:17.821+0000 10255640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-06T21:41:24.275 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:41:24.261+0000 5dc3480 -1 bdev(0x164c89d0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-02-06T21:41:24.284 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:41:24.280+0000 5dc3480 -1 bdev(0x164c89d0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-02-06T21:41:24.330 DEBUG:teuthology.orchestra.run.smithi120:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-06T21:41:24.429 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.428+0000 7f8f6b363640 10 monclient: get_monmap_and_config 2024-02-06T21:41:24.429 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.428+0000 7f8f6b363640 10 monclient: build_initial_monmap 2024-02-06T21:41:24.429 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.428+0000 7f8f6b363640 10 monclient: monmap: 2024-02-06T21:41:24.429 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:41:24.429 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:24.429 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:41:24.428608+0000 2024-02-06T21:41:24.429 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:41:24.428608+0000 2024-02-06T21:41:24.429 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:24.429 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:24.430 INFO:teuthology.misc.health.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:24.430 INFO:teuthology.misc.health.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:24.430 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:24.430 INFO:teuthology.misc.health.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:24.430 INFO:teuthology.misc.health.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:24.430 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:24.430 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.428+0000 7f8f6b363640 1 Processor -- start 2024-02-06T21:41:24.432 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 1 -- start start 2024-02-06T21:41:24.432 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 10 monclient: init 2024-02-06T21:41:24.432 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:24.432 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-02-06T21:41:24.432 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8f640b5060 0x7f8f6413f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:24.432 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 10 monclient(hunting): picked mon.noname-c con 0x7f8f640b5060 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:24.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f640a9780 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:24.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8f64142c50 0x7f8f6413fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:24.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 10 monclient(hunting): picked mon.noname-a con 0x7f8f64142c50 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:24.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f640a9d80 con 0x7f8f640a9780 2024-02-06T21:41:24.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:24.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f641401e0 con 0x7f8f64142c50 2024-02-06T21:41:24.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:24.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f640b5440 con 0x7f8f640b5060 2024-02-06T21:41:24.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:24.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.429+0000 7f8f6b363640 10 monclient(hunting): authenticate will time out at 1314.766349s 2024-02-06T21:41:24.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.430+0000 7f8f690ff640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8f640b5060 0x7f8f6413f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:24.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.430+0000 7f8f69900640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8f64142c50 0x7f8f6413fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:24.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.431+0000 7f8f690ff640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8f640b5060 0x7f8f6413f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:46872/0 (socket says 172.21.15.120:46872) 2024-02-06T21:41:24.433 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.431+0000 7f8f690ff640 1 -- 172.21.15.120:0/3754356950 learned_addr learned my addr 172.21.15.120:0/3754356950 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:24.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.431+0000 7f8f690ff640 10 monclient(hunting): get_auth_request con 0x7f8f640b5060 auth_method 0 2024-02-06T21:41:24.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.431+0000 7f8f690ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:24.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.431+0000 7f8f690ff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:24.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.431+0000 7f8f690ff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:24.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.431+0000 7f8f69900640 10 monclient(hunting): get_auth_request con 0x7f8f64142c50 auth_method 0 2024-02-06T21:41:24.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.431+0000 7f8f69900640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:24.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.431+0000 7f8f69900640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:24.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.431+0000 7f8f69900640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:24.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.431+0000 7f8f690ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:24.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.431+0000 7f8f690ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:24.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.431+0000 7f8f690ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:24.434 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.432+0000 7f8f69900640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:24.435 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.432+0000 7f8f69900640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:24.435 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.432+0000 7f8f69900640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:24.435 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.434+0000 7f8f69900640 10 monclient(hunting): handle_auth_done global_id 4233 payload 274 2024-02-06T21:41:24.436 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.434+0000 7f8f69900640 1 -- 172.21.15.120:0/3754356950 >> v2:172.21.15.135:3300/0 conn(0x7f8f640b5060 msgr2=0x7f8f6413f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:24.436 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.434+0000 7f8f69900640 1 --2- 172.21.15.120:0/3754356950 >> v2:172.21.15.135:3300/0 conn(0x7f8f640b5060 0x7f8f6413f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:24.436 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.434+0000 7f8f69900640 1 -- 172.21.15.120:0/3754356950 >> v1:172.21.15.120:6789/0 conn(0x7f8f640a9780 legacy=0x7f8f64142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:24.436 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.434+0000 7f8f69900640 10 monclient: _finish_hunting 0 2024-02-06T21:41:24.436 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.434+0000 7f8f69900640 1 monclient: found mon.noname-a 2024-02-06T21:41:24.436 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.434+0000 7f8f69900640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:24.436 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.434+0000 7f8f69900640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:24.436 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.434+0000 7f8f69900640 1 -- 172.21.15.120:0/3754356950 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f640ab070 con 0x7f8f64142c50 2024-02-06T21:41:24.438 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.438+0000 7f8f69900640 1 --2- 172.21.15.120:0/3754356950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f64142c50 0x7f8f6413fca0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8f640732a0 tx=0x7f8f5c02bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=872484c1407f78f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:24.445 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f5bfff640 1 -- 172.21.15.120:0/3754356950 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c02cb40 con 0x7f8f64142c50 2024-02-06T21:41:24.445 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:24.445 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f5bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:24.445 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f5bfff640 10 monclient: dump: 2024-02-06T21:41:24.445 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:24.445 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:24.445 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:24.445 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:24.446 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:24.446 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:24.446 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:24.446 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:24.446 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:24.446 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:24.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f5bfff640 10 monclient: _finish_auth 0 2024-02-06T21:41:24.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f5bfff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:24.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:24.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f5bfff640 1 -- 172.21.15.120:0/3754356950 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5c02cca0 con 0x7f8f64142c50 2024-02-06T21:41:24.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f6b363640 5 monclient: authenticate success, global_id 4233 2024-02-06T21:41:24.446 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f6b363640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:24.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f5bfff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:24.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f6b363640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:24.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f6b363640 4 set_mon_vals no callback set 2024-02-06T21:41:24.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f5bfff640 1 -- 172.21.15.120:0/3754356950 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c034660 con 0x7f8f64142c50 2024-02-06T21:41:24.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:24.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f5bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:24.447 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.443+0000 7f8f5bfff640 10 monclient: dump: 2024-02-06T21:41:24.447 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:24.447 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:24.447 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:24.447 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:24.448 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:24.448 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:24.448 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:24.448 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:24.448 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:24.448 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:24.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.444+0000 7f8f6b363640 10 monclient: shutdown 2024-02-06T21:41:24.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.444+0000 7f8f6b363640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:24.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.444+0000 7f8f6b363640 1 -- 172.21.15.120:0/3754356950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f64142c50 msgr2=0x7f8f6413fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:24.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.444+0000 7f8f6b363640 1 --2- 172.21.15.120:0/3754356950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f64142c50 0x7f8f6413fca0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8f640732a0 tx=0x7f8f5c02bd90 comp rx=0 tx=0).stop 2024-02-06T21:41:24.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.444+0000 7f8f6b363640 1 -- 172.21.15.120:0/3754356950 shutdown_connections 2024-02-06T21:41:24.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.444+0000 7f8f6b363640 1 --2- 172.21.15.120:0/3754356950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f64142c50 0x7f8f6413fca0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:24.448 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.444+0000 7f8f6b363640 1 --2- 172.21.15.120:0/3754356950 >> v2:172.21.15.135:3300/0 conn(0x7f8f640b5060 0x7f8f6413f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:24.449 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.444+0000 7f8f6b363640 1 -- 172.21.15.120:0/3754356950 >> 172.21.15.120:0/3754356950 conn(0x7f8f640724d0 msgr2=0x7f8f640728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:24.449 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.444+0000 7f8f6b363640 1 -- 172.21.15.120:0/3754356950 shutdown_connections 2024-02-06T21:41:24.449 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.444+0000 7f8f6b363640 1 -- 172.21.15.120:0/3754356950 wait complete. 2024-02-06T21:41:24.449 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.445+0000 7f8f6b363640 10 monclient: build_initial_monmap 2024-02-06T21:41:24.449 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.445+0000 7f8f6b363640 10 monclient: monmap: 2024-02-06T21:41:24.449 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:41:24.449 INFO:teuthology.misc.health.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:24.449 INFO:teuthology.misc.health.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:24.449 INFO:teuthology.misc.health.smithi120.stderr:created 0.000000 2024-02-06T21:41:24.449 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:24.449 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:24.449 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:24.449 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:24.450 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:24.450 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:24.452 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.445+0000 7f8f6b363640 1 librados: starting msgr at 2024-02-06T21:41:24.452 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.445+0000 7f8f6b363640 1 librados: starting objecter 2024-02-06T21:41:24.452 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.445+0000 7f8f6b363640 1 Processor -- start 2024-02-06T21:41:24.452 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.445+0000 7f8f6b363640 1 -- start start 2024-02-06T21:41:24.452 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.445+0000 7f8f6b363640 1 librados: setting wanted keys 2024-02-06T21:41:24.452 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.445+0000 7f8f6b363640 1 librados: calling monclient init 2024-02-06T21:41:24.452 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.445+0000 7f8f6b363640 10 monclient: init 2024-02-06T21:41:24.452 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:24.452 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T21:41:24.452 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f640a9780 0x7f8f64138ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:24.452 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 10 monclient(hunting): picked mon.noname-b con 0x7f8f640a9780 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:24.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f640b5060 0x7f8f64139420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:24.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 10 monclient(hunting): picked mon.noname-a con 0x7f8f640b5060 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:24.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f64142c50 0x7f8f641d74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:24.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 10 monclient(hunting): picked mon.noname-c con 0x7f8f64142c50 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:24.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:24.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f64142670 con 0x7f8f640b5060 2024-02-06T21:41:24.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:24.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f641421e0 con 0x7f8f640a9780 2024-02-06T21:41:24.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:24.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f64138470 con 0x7f8f64142c50 2024-02-06T21:41:24.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:24.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f6b363640 10 monclient(hunting): authenticate will time out at 1314.782932s 2024-02-06T21:41:24.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f688fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f640b5060 0x7f8f64139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:24.453 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f690ff640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f640a9780 0x7f8f64138ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:24.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.446+0000 7f8f69900640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f64142c50 0x7f8f641d74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:24.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f69900640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f64142c50 0x7f8f641d74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:54678/0 (socket says 172.21.15.120:54678) 2024-02-06T21:41:24.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f69900640 1 -- 172.21.15.120:0/3233750503 learned_addr learned my addr 172.21.15.120:0/3233750503 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:24.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f69900640 10 monclient(hunting): get_auth_request con 0x7f8f64142c50 auth_method 0 2024-02-06T21:41:24.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f69900640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:24.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f69900640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:24.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f69900640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:24.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f690ff640 10 monclient(hunting): get_auth_request con 0x7f8f640a9780 auth_method 0 2024-02-06T21:41:24.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f690ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:24.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f690ff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:24.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f690ff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:24.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f688fe640 10 monclient(hunting): get_auth_request con 0x7f8f640b5060 auth_method 0 2024-02-06T21:41:24.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f688fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:24.454 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f688fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:24.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f688fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:24.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f69900640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:24.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f69900640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:24.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.447+0000 7f8f69900640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:24.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.448+0000 7f8f690ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:24.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.448+0000 7f8f690ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:24.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.448+0000 7f8f690ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:24.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.448+0000 7f8f688fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:24.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.448+0000 7f8f688fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:24.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.448+0000 7f8f688fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:24.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.450+0000 7f8f69900640 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2024-02-06T21:41:24.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.451+0000 7f8f69900640 1 -- 172.21.15.120:0/3233750503 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f640a9780 msgr2=0x7f8f64138ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:24.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.451+0000 7f8f69900640 1 --2- 172.21.15.120:0/3233750503 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f640a9780 0x7f8f64138ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:24.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.451+0000 7f8f69900640 1 -- 172.21.15.120:0/3233750503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f640b5060 msgr2=0x7f8f64139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:24.455 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.451+0000 7f8f69900640 1 --2- 172.21.15.120:0/3233750503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f640b5060 0x7f8f64139420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:24.456 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.451+0000 7f8f69900640 10 monclient: _finish_hunting 0 2024-02-06T21:41:24.456 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.451+0000 7f8f69900640 1 monclient: found mon.noname-c 2024-02-06T21:41:24.456 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.451+0000 7f8f69900640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:24.456 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.451+0000 7f8f69900640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:41:24.456 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.451+0000 7f8f69900640 1 -- 172.21.15.120:0/3233750503 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c009a70 con 0x7f8f64142c50 2024-02-06T21:41:24.456 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.454+0000 7f8f69900640 1 --2- 172.21.15.120:0/3233750503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f64142c50 0x7f8f641d74f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c02c3a0 tx=0x7f8f5c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:24.456 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.454+0000 7f8f59ffb640 10 client.?.objecter ms_handle_connect 0x7f8f64142c50 2024-02-06T21:41:24.456 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.454+0000 7f8f59ffb640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:24.461 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.458+0000 7f8f59ffb640 1 -- 172.21.15.120:0/3233750503 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c03e600 con 0x7f8f64142c50 2024-02-06T21:41:24.461 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.458+0000 7f8f59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:24.461 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.458+0000 7f8f59ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:24.461 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f59ffb640 10 monclient: dump: 2024-02-06T21:41:24.461 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:24.461 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:24.461 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:24.461 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:24.461 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:24.461 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:24.462 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:24.462 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:24.462 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:24.462 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:24.462 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f59ffb640 10 monclient: _finish_auth 0 2024-02-06T21:41:24.462 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f59ffb640 10 monclient: _check_auth_tickets 2024-02-06T21:41:24.462 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:24.462 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f59ffb640 1 -- 172.21.15.120:0/3233750503 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5c03e760 con 0x7f8f64142c50 2024-02-06T21:41:24.462 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f6b363640 5 monclient: authenticate success, global_id 4229 2024-02-06T21:41:24.462 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f6b363640 10 monclient: _renew_subs 2024-02-06T21:41:24.462 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f6b363640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:24.462 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f6b363640 1 -- 172.21.15.120:0/3233750503 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f64138750 con 0x7f8f64142c50 2024-02-06T21:41:24.462 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f59ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:24.463 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f59ffb640 1 -- 172.21.15.120:0/3233750503 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c03eb10 con 0x7f8f64142c50 2024-02-06T21:41:24.463 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:24.463 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f59ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:24.463 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f59ffb640 10 monclient: dump: 2024-02-06T21:41:24.463 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:24.463 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:24.463 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:24.463 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:24.463 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:24.463 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:24.463 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:24.463 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:24.463 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:24.464 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:24.464 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f6b363640 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:24.464 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f6b363640 10 monclient: _renew_subs 2024-02-06T21:41:24.464 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f6b363640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:24.464 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f6b363640 1 -- 172.21.15.120:0/3233750503 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f64141e80 con 0x7f8f64142c50 2024-02-06T21:41:24.464 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f6b363640 1 librados: init done 2024-02-06T21:41:24.464 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.459+0000 7f8f5b7fe640 4 set_mon_vals no callback set 2024-02-06T21:41:24.464 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.460+0000 7f8f6b363640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:24.464 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.460+0000 7f8f6b363640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:24.464 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.460+0000 7f8f6b363640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:24.464 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.460+0000 7f8f6b363640 1 -- 172.21.15.120:0/3233750503 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6413f0e0 con 0x7f8f64142c50 2024-02-06T21:41:24.500 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:41:24.340+0000 5dc3480 -1 bdev(0x164d0570 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-02-06T21:41:24.501 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:41:24.354+0000 5dc3480 -1 bdev(0x164d0570 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-02-06T21:41:24.559 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:41:24.500+0000 5dc3480 -1 bdev(0x164d0570 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 3 2024-02-06T21:41:24.559 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:41:24.535+0000 5dc3480 -1 bdev(0x164d0570 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-02-06T21:41:24.599 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:24.517+0000 5dc3480 -1 bdev(0x164ccba0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-02-06T21:41:24.616 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:41:24.560+0000 5dc3480 -1 bdev(0x164d0570 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 2 2024-02-06T21:41:24.621 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:41:24.550+0000 5dc3480 -1 bdev(0x164cf2c0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-02-06T21:41:24.621 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:41:24.564+0000 5dc3480 -1 bdev(0x164cf2c0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-02-06T21:41:24.674 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:41:24.621+0000 5dc3480 -1 bdev(0x164cf2c0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-02-06T21:41:24.685 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:41:24.662+0000 5dc3480 -1 bdev(0x164cf2c0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-02-06T21:41:24.723 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.723+0000 7f8f59ffb640 1 -- 172.21.15.120:0/3233750503 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 89732+0+0 (secure 0 0 0) 0x7f8f5c03ec70 con 0x7f8f64142c50 2024-02-06T21:41:24.723 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.723+0000 7f8f59ffb640 1 --2- 172.21.15.120:0/3233750503 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8f4806d160 0x7f8f4806f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:24.724 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.723+0000 7f8f690ff640 1 --2- 172.21.15.120:0/3233750503 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8f4806d160 0x7f8f4806f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:24.724 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.723+0000 7f8f690ff640 10 monclient: get_auth_request con 0x7f8f4806d160 auth_method 0 2024-02-06T21:41:24.725 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.724+0000 7f8f690ff640 1 --2- 172.21.15.120:0/3233750503 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8f4806d160 0x7f8f4806f620 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009bd0 tx=0x7f8f60009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:24.725 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.724+0000 7f8f59ffb640 10 client.4229.objecter ms_handle_connect 0x7f8f4806d160 2024-02-06T21:41:24.736 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.735+0000 7f8f59ffb640 1 -- 172.21.15.120:0/3233750503 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3177+0+0 (secure 0 0 0) 0x7f8f5c07baf0 con 0x7f8f64142c50 2024-02-06T21:41:24.736 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.735+0000 7f8f59ffb640 10 client.4229.objecter ms_dispatch 0x7f8f64000b90 osd_map(16..16 src has 1..16) v4 2024-02-06T21:41:24.736 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.735+0000 7f8f59ffb640 3 client.4229.objecter handle_osd_map got epochs [16,16] > 0 2024-02-06T21:41:24.736 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.735+0000 7f8f59ffb640 3 client.4229.objecter handle_osd_map decoding full epoch 16 2024-02-06T21:41:24.736 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.735+0000 7f8f59ffb640 20 client.4229.objecter dump_active .. 0 homeless 2024-02-06T21:41:24.736 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.735+0000 7f8f59ffb640 1 -- 172.21.15.120:0/3233750503 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8f5c077d90 con 0x7f8f64142c50 2024-02-06T21:41:24.736 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.735+0000 7f8f59ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:24.737 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.735+0000 7f8f59ffb640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:24.743 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:24.599+0000 5dc3480 -1 bdev(0x164ccba0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-02-06T21:41:24.743 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:24.731+0000 5dc3480 -1 bdev(0x164ccba0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-02-06T21:41:24.766 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:24.743+0000 5dc3480 -1 bdev(0x164ccba0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-02-06T21:41:24.778 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:24.754+0000 5dc3480 -1 bdev(0x164ccba0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-02-06T21:41:24.790 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:24.766+0000 5dc3480 -1 bdev(0x164ccba0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-02-06T21:41:24.791 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:24.777+0000 5dc3480 -1 bdev(0x164ccba0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-02-06T21:41:24.841 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:24.790+0000 5dc3480 -1 bdev(0x164ccba0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-02-06T21:41:24.928 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.927+0000 7f8f6b363640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:24.928 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.927+0000 7f8f6b363640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:24.928 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.927+0000 7f8f6b363640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:24.928 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:24.927+0000 7f8f6b363640 1 -- 172.21.15.120:0/3233750503 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f641385b0 con 0x7f8f64142c50 2024-02-06T21:41:24.958 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:24.955+0000 5dc3480 -1 bdev(0x164ccba0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-02-06T21:41:25.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.000+0000 7f8f59ffb640 1 -- 172.21.15.120:0/3233750503 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+9658 (secure 0 0 0) 0x7f8f641385b0 con 0x7f8f64142c50 2024-02-06T21:41:25.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.000+0000 7f8f59ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:25.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.000+0000 7f8f59ffb640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:25.002 INFO:teuthology.misc.health.smithi120.stdout: 2024-02-06T21:41:25.002 INFO:teuthology.misc.health.smithi120.stdout:{"epoch":16,"fsid":"4e4791bf-0a02-4478-9517-2d2e893f4f7c","created":"2024-02-06T21:40:31.800075+0000","modified":"2024-02-06T21:41:21.611803+0000","last_up_change":"2024-02-06T21:41:18.074865+0000","last_in_change":"2024-02-06T21:40:37.354288+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-06T21:41:20.093970+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4bd0f591-2cf1-42da-8ac1-dce3188580c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6817","nonce":2062960776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6819","nonce":2062960776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6823","nonce":2062960776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6821","nonce":2062960776}]},"public_addr":"172.21.15.120:6817/2062960776","cluster_addr":"172.21.15.120:6819/2062960776","heartbeat_back_addr":"172.21.15.120:6823/2062960776","heartbeat_front_addr":"172.21.15.120:6821/2062960776","state":["exists","up"]},{"osd":1,"uuid":"81dc801c-3768-4c2c-9adb-6c8300a341f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9289721b-c13f-4a17-be68-3698090f69af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"53534db0-2d4d-44ed-90e8-d67ea878b51a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e4205ab5-787c-44e0-9b96-273b8c9cf1ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"43b600bc-3afe-44bd-bd3f-690106e7e0ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1d6b9f96-1f9d-44f2-83b0-93f5c342fafb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecae00f5-8e29-47dd-90d5-e51503bc9dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6825","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6827","nonce":3740758337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6829","nonce":3740758337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6833","nonce":3740758337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6831","nonce":3740758337}]},"public_addr":"172.21.15.135:6827/3740758337","cluster_addr":"172.21.15.135:6829/3740758337","heartbeat_back_addr":"172.21.15.135:6833/3740758337","heartbeat_front_addr":"172.21.15.135:6831/3740758337","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.344503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:15.753628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-06T21:41:25.004 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.003+0000 7f8f6b363640 10 librados: watch_flush enter 2024-02-06T21:41:25.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.003+0000 7f8f6b363640 10 librados: watch_flush exit 2024-02-06T21:41:25.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.003+0000 7f8f6b363640 20 client.4229.objecter shutdown clearing up homeless session... 2024-02-06T21:41:25.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.003+0000 7f8f6b363640 10 client.4229.objecter successfully canceled tick 2024-02-06T21:41:25.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.004+0000 7f8f6b363640 1 -- 172.21.15.120:0/3233750503 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8f4806d160 msgr2=0x7f8f4806f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:25.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.004+0000 7f8f6b363640 1 --2- 172.21.15.120:0/3233750503 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8f4806d160 0x7f8f4806f620 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009bd0 tx=0x7f8f60009240 comp rx=0 tx=0).stop 2024-02-06T21:41:25.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.004+0000 7f8f6b363640 10 monclient: shutdown 2024-02-06T21:41:25.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.004+0000 7f8f6b363640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:25.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.004+0000 7f8f6b363640 1 -- 172.21.15.120:0/3233750503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f64142c50 msgr2=0x7f8f641d74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:25.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.004+0000 7f8f6b363640 1 --2- 172.21.15.120:0/3233750503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f64142c50 0x7f8f641d74f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c02c3a0 tx=0x7f8f5c004a40 comp rx=0 tx=0).stop 2024-02-06T21:41:25.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.004+0000 7f8f6b363640 1 -- 172.21.15.120:0/3233750503 shutdown_connections 2024-02-06T21:41:25.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.004+0000 7f8f6b363640 1 --2- 172.21.15.120:0/3233750503 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f8f4806d160 0x7f8f4806f620 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:25.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.004+0000 7f8f6b363640 1 --2- 172.21.15.120:0/3233750503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f64142c50 0x7f8f641d74f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:25.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.004+0000 7f8f6b363640 1 --2- 172.21.15.120:0/3233750503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f640b5060 0x7f8f64139420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:25.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.004+0000 7f8f6b363640 1 --2- 172.21.15.120:0/3233750503 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f640a9780 0x7f8f64138ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:25.006 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.004+0000 7f8f6b363640 1 -- 172.21.15.120:0/3233750503 >> 172.21.15.120:0/3233750503 conn(0x7f8f640724d0 msgr2=0x7f8f640728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:25.006 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.004+0000 7f8f6b363640 1 -- 172.21.15.120:0/3233750503 shutdown_connections 2024-02-06T21:41:25.006 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.004+0000 7f8f6b363640 1 -- 172.21.15.120:0/3233750503 wait complete. 2024-02-06T21:41:25.006 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:25.004+0000 7f8f6b363640 1 librados: shutdown 2024-02-06T21:41:25.021 DEBUG:teuthology.misc:2 of 8 OSDs are up 2024-02-06T21:41:25.077 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:41:24.641+0000 5dc3480 -1 bdev(0x164c89d0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-02-06T21:41:25.077 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:41:24.669+0000 5dc3480 -1 bdev(0x164c89d0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-02-06T21:41:25.204 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:41:24.630+0000 5dc3480 -1 bdev(0x164d0570 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 2 2024-02-06T21:41:25.224 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:24.959+0000 5dc3480 -1 bdev(0x164ccba0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-02-06T21:41:25.224 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:25.067+0000 5dc3480 -1 bdev(0x164ccba0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-02-06T21:41:25.272 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:25.223+0000 5dc3480 -1 bdev(0x164ccba0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-02-06T21:41:25.424 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:41:24.685+0000 5dc3480 -1 bdev(0x164cf2c0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-02-06T21:41:25.441 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:25.272+0000 5dc3480 -1 bdev(0x164ccba0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-02-06T21:41:25.461 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:41:24.646+0000 5dc3480 -1 bdev(0x18efc0f0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-02-06T21:41:26.220 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:41:26.215+0000 c48f640 -1 osd.4 0 waiting for initial osdmap 2024-02-06T21:41:26.255 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:41:26.249+0000 ac79640 -1 osd.5 0 waiting for initial osdmap 2024-02-06T21:41:26.299 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:25.927+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 3 2024-02-06T21:41:26.299 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:26.178+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-02-06T21:41:26.312 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:26.280+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-02-06T21:41:26.348 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:26.299+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-02-06T21:41:26.348 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:26.312+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-02-06T21:41:26.348 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:26.324+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-02-06T21:41:26.474 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:26.348+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-02-06T21:41:26.474 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:26.432+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-02-06T21:41:26.491 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:26.473+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-02-06T21:41:26.498 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:41:26.493+0000 ac79640 -1 osd.2 0 waiting for initial osdmap 2024-02-06T21:41:26.585 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:26.581+0000 c48f640 -1 osd.6 0 waiting for initial osdmap 2024-02-06T21:41:26.586 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:41:26.580+0000 ac79640 -1 osd.3 0 waiting for initial osdmap 2024-02-06T21:41:26.641 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:26.491+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-02-06T21:41:26.641 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:26.505+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-02-06T21:41:26.665 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:26.640+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-02-06T21:41:26.689 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:26.666+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-02-06T21:41:26.700 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:41:26.685+0000 1425d640 -1 osd.4 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-06T21:41:26.702 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:26.689+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-02-06T21:41:26.742 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:26.704+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-02-06T21:41:26.773 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:41:26.759+0000 1425d640 -1 osd.5 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-06T21:41:26.877 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:26.876+0000 5dc3480 -1 bdev(0x164bf090 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-02-06T21:41:27.058 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:41:27.037+0000 1425d640 -1 osd.6 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-06T21:41:27.097 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:41:27.081+0000 1425d640 -1 osd.3 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-06T21:41:27.128 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:41:27.112+0000 1425d640 -1 osd.2 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-06T21:41:28.033 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:27.998+0000 ac79640 -1 osd.1 0 waiting for initial osdmap 2024-02-06T21:41:28.556 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:41:28.542+0000 1425d640 -1 osd.1 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-06T21:41:29.459 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:01:09.274 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:29.528 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:01:09.342 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:29.529 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:01:09.293 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:29.561 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:01:09.376 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:29.592 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:01:09.356 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:29.616 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:01:09.430 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:29.791 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:01:09.605 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:29.814 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:01:09.579 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:29.851 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:01:09.666 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:31.826 DEBUG:teuthology.orchestra.run.smithi120:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-06T21:41:31.929 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.928+0000 7f5fc4964640 10 monclient: get_monmap_and_config 2024-02-06T21:41:31.929 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.928+0000 7f5fc4964640 10 monclient: build_initial_monmap 2024-02-06T21:41:31.929 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.928+0000 7f5fc4964640 10 monclient: monmap: 2024-02-06T21:41:31.929 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:41:31.929 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:31.929 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:41:31.928854+0000 2024-02-06T21:41:31.929 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:41:31.928854+0000 2024-02-06T21:41:31.929 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:31.929 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:31.929 INFO:teuthology.misc.health.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:31.929 INFO:teuthology.misc.health.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:31.930 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:31.930 INFO:teuthology.misc.health.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:31.930 INFO:teuthology.misc.health.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:31.930 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:31.931 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.929+0000 7f5fc4964640 1 Processor -- start 2024-02-06T21:41:31.931 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.929+0000 7f5fc4964640 1 -- start start 2024-02-06T21:41:31.931 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.929+0000 7f5fc4964640 10 monclient: init 2024-02-06T21:41:31.931 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.930+0000 7f5fc4964640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:31.931 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.930+0000 7f5fc4964640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-02-06T21:41:31.931 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.930+0000 7f5fc4964640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5fc00b5060 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:31.931 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.930+0000 7f5fc4964640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5fc013fe70 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:31.931 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.930+0000 7f5fc4964640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5fc00a9780 0x7f5fc014de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:31.932 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.930+0000 7f5fc4964640 10 monclient(hunting): picked mon.noname-a con 0x7f5fc00a9780 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:31.932 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.930+0000 7f5fc4964640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc00a9d80 con 0x7f5fc00b5060 2024-02-06T21:41:31.932 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.930+0000 7f5fc4964640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc00ab070 con 0x7f5fc013fe70 2024-02-06T21:41:31.932 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.930+0000 7f5fc4964640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:31.932 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.930+0000 7f5fc4964640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc00b5440 con 0x7f5fc00a9780 2024-02-06T21:41:31.932 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.930+0000 7f5fc4964640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:31.932 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.930+0000 7f5fc4964640 10 monclient(hunting): authenticate will time out at 1322.267016s 2024-02-06T21:41:31.975 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.974+0000 7f5fbdd9b640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f5fc013fe70 0x7f5fc0142540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.120:41904/0 (socket says 172.21.15.120:41904) 2024-02-06T21:41:31.975 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.974+0000 7f5fbdd9b640 1 -- 172.21.15.120:0/1232613029 learned_addr learned my addr 172.21.15.120:0/1232613029 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:41:31.980 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.980+0000 7f5fbed9d640 1 --2- 172.21.15.120:0/1232613029 >> v2:172.21.15.120:3300/0 conn(0x7f5fc00a9780 0x7f5fc014de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:31.981 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.980+0000 7f5fbd59a640 1 -- 172.21.15.120:0/1232613029 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716658012 0 0) 0x7f5fc00ab070 con 0x7f5fc013fe70 2024-02-06T21:41:31.981 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.980+0000 7f5fbd59a640 10 monclient(hunting): _negotiate 2024-02-06T21:41:31.981 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.980+0000 7f5fbd59a640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:31.981 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.980+0000 7f5fbd59a640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:31.981 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.980+0000 7f5fbd59a640 10 monclient(hunting): my global_id is 4246 2024-02-06T21:41:31.981 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.980+0000 7f5fbd59a640 1 -- 172.21.15.120:0/1232613029 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fa8003520 con 0x7f5fc013fe70 2024-02-06T21:41:31.981 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.981+0000 7f5fbed9d640 10 monclient(hunting): get_auth_request con 0x7f5fc00a9780 auth_method 0 2024-02-06T21:41:31.982 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.981+0000 7f5fbed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:31.982 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.981+0000 7f5fbed9d640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:31.982 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.981+0000 7f5fbed9d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:31.983 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.982+0000 7f5fbed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:31.983 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.982+0000 7f5fbed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:31.983 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.982+0000 7f5fbed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:31.985 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.984+0000 7f5fbed9d640 10 monclient(hunting): handle_auth_done global_id 4239 payload 274 2024-02-06T21:41:31.985 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.985+0000 7f5fbed9d640 1 -- 172.21.15.120:0/1232613029 >> v1:172.21.15.135:6789/0 conn(0x7f5fc013fe70 legacy=0x7f5fc0142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:31.985 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.985+0000 7f5fbed9d640 1 -- 172.21.15.120:0/1232613029 >> v1:172.21.15.120:6789/0 conn(0x7f5fc00b5060 legacy=0x7f5fc013f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:31.985 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.985+0000 7f5fbed9d640 10 monclient: _finish_hunting 0 2024-02-06T21:41:31.985 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.985+0000 7f5fbed9d640 1 monclient: found mon.noname-a 2024-02-06T21:41:31.986 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.985+0000 7f5fbed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:31.986 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.985+0000 7f5fbed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:31.986 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.985+0000 7f5fbed9d640 1 -- 172.21.15.120:0/1232613029 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc00ab350 con 0x7f5fc00a9780 2024-02-06T21:41:31.987 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.987+0000 7f5fbed9d640 1 --2- 172.21.15.120:0/1232613029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fc00a9780 0x7f5fc014de40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8009d90 tx=0x7f5fb802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c936fc5fcd0d30fd server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:31.992 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.991+0000 7f5fbd59a640 1 -- 172.21.15.120:0/1232613029 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb802cbd0 con 0x7f5fc00a9780 2024-02-06T21:41:31.992 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.991+0000 7f5fbd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:31.992 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.991+0000 7f5fbd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:31.992 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.991+0000 7f5fbd59a640 10 monclient: dump: 2024-02-06T21:41:31.992 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:31.992 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:31.992 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:31.993 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:31.993 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:31.993 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:31.993 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:31.993 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:31.993 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:31.993 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:31.993 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.991+0000 7f5fbd59a640 10 monclient: _finish_auth 0 2024-02-06T21:41:31.993 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.991+0000 7f5fbd59a640 10 monclient: _check_auth_tickets 2024-02-06T21:41:31.993 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.991+0000 7f5fbd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:31.993 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.991+0000 7f5fc4964640 5 monclient: authenticate success, global_id 4239 2024-02-06T21:41:31.993 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.991+0000 7f5fc4964640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:31.994 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.994+0000 7f5fbd59a640 1 -- 172.21.15.120:0/1232613029 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb802cd30 con 0x7f5fc00a9780 2024-02-06T21:41:31.994 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.994+0000 7f5fbd59a640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:31.994 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.994+0000 7f5fbd59a640 1 -- 172.21.15.120:0/1232613029 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb802cea0 con 0x7f5fc00a9780 2024-02-06T21:41:31.995 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.994+0000 7f5fc4964640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:31.995 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.994+0000 7f5fc4964640 4 set_mon_vals no callback set 2024-02-06T21:41:31.995 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.994+0000 7f5fbd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:31.995 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.994+0000 7f5fbd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:31.995 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.994+0000 7f5fbd59a640 10 monclient: dump: 2024-02-06T21:41:31.995 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:31.995 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:31.995 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:31.995 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:31.995 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:31.995 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:31.995 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:31.995 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:31.995 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:31.995 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:31.998 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.994+0000 7f5fc4964640 10 monclient: shutdown 2024-02-06T21:41:31.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.994+0000 7f5fc4964640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:31.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.994+0000 7f5fc4964640 1 -- 172.21.15.120:0/1232613029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fc00a9780 msgr2=0x7f5fc014de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:31.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.994+0000 7f5fc4964640 1 --2- 172.21.15.120:0/1232613029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fc00a9780 0x7f5fc014de40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8009d90 tx=0x7f5fb802be20 comp rx=0 tx=0).stop 2024-02-06T21:41:31.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.995+0000 7f5fc4964640 1 -- 172.21.15.120:0/1232613029 shutdown_connections 2024-02-06T21:41:31.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.995+0000 7f5fc4964640 1 --2- 172.21.15.120:0/1232613029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fc00a9780 0x7f5fc014de40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:31.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.995+0000 7f5fc4964640 1 -- 172.21.15.120:0/1232613029 >> 172.21.15.120:0/1232613029 conn(0x7f5fc00724d0 msgr2=0x7f5fc00728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:31.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.995+0000 7f5fc4964640 1 -- 172.21.15.120:0/1232613029 shutdown_connections 2024-02-06T21:41:31.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.995+0000 7f5fc4964640 1 -- 172.21.15.120:0/1232613029 wait complete. 2024-02-06T21:41:31.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.995+0000 7f5fc4964640 10 monclient: build_initial_monmap 2024-02-06T21:41:31.999 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.995+0000 7f5fc4964640 10 monclient: monmap: 2024-02-06T21:41:31.999 INFO:teuthology.misc.health.smithi120.stderr:epoch 0 2024-02-06T21:41:31.999 INFO:teuthology.misc.health.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:31.999 INFO:teuthology.misc.health.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:31.999 INFO:teuthology.misc.health.smithi120.stderr:created 0.000000 2024-02-06T21:41:32.000 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:32.000 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:32.000 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:32.000 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:32.000 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:32.000 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:32.000 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.996+0000 7f5fc4964640 1 librados: starting msgr at 2024-02-06T21:41:32.000 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.996+0000 7f5fc4964640 1 librados: starting objecter 2024-02-06T21:41:32.000 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.996+0000 7f5fc4964640 1 Processor -- start 2024-02-06T21:41:32.000 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.996+0000 7f5fc4964640 1 -- start start 2024-02-06T21:41:32.000 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.996+0000 7f5fc4964640 1 librados: setting wanted keys 2024-02-06T21:41:32.000 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.996+0000 7f5fc4964640 1 librados: calling monclient init 2024-02-06T21:41:32.000 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.996+0000 7f5fc4964640 10 monclient: init 2024-02-06T21:41:32.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.996+0000 7f5fc4964640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:32.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fc4964640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T21:41:32.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fc4964640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fc00a9780 0x7f5fc01464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:32.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fc4964640 10 monclient(hunting): picked mon.noname-b con 0x7f5fc00a9780 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:32.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fc4964640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fc00b5060 0x7f5fc00abd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:32.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fc4964640 10 monclient(hunting): picked mon.noname-c con 0x7f5fc00b5060 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:32.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fc4964640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fc013fe70 0x7f5fc0138360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:32.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fc4964640 10 monclient(hunting): picked mon.noname-a con 0x7f5fc013fe70 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:32.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fc4964640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:32.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fc4964640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc00ad910 con 0x7f5fc013fe70 2024-02-06T21:41:32.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fc4964640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:32.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fc4964640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc00ac700 con 0x7f5fc00a9780 2024-02-06T21:41:32.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fc4964640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:32.001 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fc4964640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc00ac290 con 0x7f5fc00b5060 2024-02-06T21:41:32.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fc4964640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:32.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fc4964640 10 monclient(hunting): authenticate will time out at 1322.333601s 2024-02-06T21:41:32.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fbed9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fc013fe70 0x7f5fc0138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:32.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fbed9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fc013fe70 0x7f5fc0138360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:32986/0 (socket says 172.21.15.120:32986) 2024-02-06T21:41:32.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fbed9d640 1 -- 172.21.15.120:0/1353137138 learned_addr learned my addr 172.21.15.120:0/1353137138 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:32.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fbed9d640 10 monclient(hunting): get_auth_request con 0x7f5fc013fe70 auth_method 0 2024-02-06T21:41:32.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fbed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:32.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fbe59c640 1 --2- 172.21.15.120:0/1353137138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fc00a9780 0x7f5fc01464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:32.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fbed9d640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:32.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.997+0000 7f5fbed9d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:32.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.998+0000 7f5fbe59c640 10 monclient(hunting): get_auth_request con 0x7f5fc00a9780 auth_method 0 2024-02-06T21:41:32.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.998+0000 7f5fbe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:32.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.998+0000 7f5fbe59c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:32.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.998+0000 7f5fbe59c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:32.002 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.998+0000 7f5fbed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:32.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.998+0000 7f5fbed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:32.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.998+0000 7f5fbed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:32.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.998+0000 7f5fbe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:32.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.998+0000 7f5fbe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:32.003 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:31.998+0000 7f5fbe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:32.004 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.001+0000 7f5fbe59c640 10 monclient(hunting): handle_auth_done global_id 4249 payload 824 2024-02-06T21:41:32.004 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.001+0000 7f5fbe59c640 1 -- 172.21.15.120:0/1353137138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fc00b5060 msgr2=0x7f5fc00abd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:32.004 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.001+0000 7f5fbe59c640 1 --2- 172.21.15.120:0/1353137138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fc00b5060 0x7f5fc00abd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:32.004 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.001+0000 7f5fbe59c640 1 -- 172.21.15.120:0/1353137138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fc013fe70 msgr2=0x7f5fc0138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:32.004 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.001+0000 7f5fbe59c640 1 --2- 172.21.15.120:0/1353137138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fc013fe70 0x7f5fc0138360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:32.004 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.001+0000 7f5fbe59c640 10 monclient: _finish_hunting 0 2024-02-06T21:41:32.004 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.001+0000 7f5fbe59c640 1 monclient: found mon.noname-b 2024-02-06T21:41:32.004 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.001+0000 7f5fbe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:32.004 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.001+0000 7f5fbe59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:32.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.001+0000 7f5fbe59c640 1 -- 172.21.15.120:0/1353137138 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb8009a70 con 0x7f5fc00a9780 2024-02-06T21:41:32.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.001+0000 7f5fbed9d640 1 --2- 172.21.15.120:0/1353137138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fc013fe70 0x7f5fc0138360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-06T21:41:32.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.003+0000 7f5fbe59c640 1 --2- 172.21.15.120:0/1353137138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fc00a9780 0x7f5fc01464a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000c410 tx=0x7f5fb000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:32.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.003+0000 7f5faeffd640 10 client.?.objecter ms_handle_connect 0x7f5fc00a9780 2024-02-06T21:41:32.005 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.003+0000 7f5faeffd640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:32.008 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5faeffd640 1 -- 172.21.15.120:0/1353137138 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb0005230 con 0x7f5fc00a9780 2024-02-06T21:41:32.008 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5faeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:32.008 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5faeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:32.008 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5faeffd640 10 monclient: dump: 2024-02-06T21:41:32.008 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:32.008 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:32.008 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:32.008 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:32.008 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:32.008 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:32.008 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:32.009 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:32.009 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:32.009 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:32.009 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5faeffd640 10 monclient: _finish_auth 0 2024-02-06T21:41:32.009 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5faeffd640 10 monclient: _check_auth_tickets 2024-02-06T21:41:32.009 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5faeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:32.009 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5faeffd640 1 -- 172.21.15.120:0/1353137138 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb0005390 con 0x7f5fc00a9780 2024-02-06T21:41:32.009 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5faeffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:32.009 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5faeffd640 1 -- 172.21.15.120:0/1353137138 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb0004820 con 0x7f5fc00a9780 2024-02-06T21:41:32.009 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5faeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:32.009 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5faeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:32.009 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5faeffd640 10 monclient: dump: 2024-02-06T21:41:32.009 INFO:teuthology.misc.health.smithi120.stderr:epoch 1 2024-02-06T21:41:32.009 INFO:teuthology.misc.health.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:32.010 INFO:teuthology.misc.health.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:32.010 INFO:teuthology.misc.health.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:32.010 INFO:teuthology.misc.health.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:32.010 INFO:teuthology.misc.health.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:32.010 INFO:teuthology.misc.health.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:32.010 INFO:teuthology.misc.health.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:32.010 INFO:teuthology.misc.health.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:32.010 INFO:teuthology.misc.health.smithi120.stderr: 2024-02-06T21:41:32.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5fc4964640 5 monclient: authenticate success, global_id 4249 2024-02-06T21:41:32.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5fc4964640 10 monclient: _renew_subs 2024-02-06T21:41:32.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5fbcd99640 4 set_mon_vals no callback set 2024-02-06T21:41:32.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5fc4964640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:32.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5fc4964640 1 -- 172.21.15.120:0/1353137138 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc01388a0 con 0x7f5fc00a9780 2024-02-06T21:41:32.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5fc4964640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:32.010 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5fc4964640 10 monclient: _renew_subs 2024-02-06T21:41:32.011 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5fc4964640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:32.011 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5fc4964640 1 -- 172.21.15.120:0/1353137138 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc0138d60 con 0x7f5fc00a9780 2024-02-06T21:41:32.011 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.007+0000 7f5fc4964640 1 librados: init done 2024-02-06T21:41:32.011 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.008+0000 7f5fc4964640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:32.011 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.008+0000 7f5fc4964640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:32.011 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.008+0000 7f5fc4964640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:32.011 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.008+0000 7f5fc4964640 1 -- 172.21.15.120:0/1353137138 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc00aa1f0 con 0x7f5fc00a9780 2024-02-06T21:41:32.027 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.026+0000 7f5faeffd640 1 -- 172.21.15.120:0/1353137138 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 4) v1 ==== 89732+0+0 (secure 0 0 0) 0x7f5fb0004980 con 0x7f5fc00a9780 2024-02-06T21:41:32.027 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.026+0000 7f5faeffd640 1 --2- 172.21.15.120:0/1353137138 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f5fa006d110 0x7f5fa006f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:32.027 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.026+0000 7f5fbdd9b640 1 --2- 172.21.15.120:0/1353137138 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f5fa006d110 0x7f5fa006f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:32.027 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.026+0000 7f5fbdd9b640 10 monclient: get_auth_request con 0x7f5fa006d110 auth_method 0 2024-02-06T21:41:32.027 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.027+0000 7f5faeffd640 10 client.4249.objecter ms_handle_connect 0x7f5fa006d110 2024-02-06T21:41:32.027 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.027+0000 7f5fbdd9b640 1 --2- 172.21.15.120:0/1353137138 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f5fa006d110 0x7f5fa006f5d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0145770 tx=0x7f5fb4007930 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:32.029 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.028+0000 7f5faeffd640 1 -- 172.21.15.120:0/1353137138 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5fb0057e30 con 0x7f5fc00a9780 2024-02-06T21:41:32.029 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.028+0000 7f5faeffd640 10 client.4249.objecter ms_dispatch 0x7f5fc0000b90 osd_map(19..19 src has 1..19) v4 2024-02-06T21:41:32.029 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.028+0000 7f5faeffd640 3 client.4249.objecter handle_osd_map got epochs [19,19] > 0 2024-02-06T21:41:32.029 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.028+0000 7f5faeffd640 3 client.4249.objecter handle_osd_map decoding full epoch 19 2024-02-06T21:41:32.029 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.028+0000 7f5faeffd640 20 client.4249.objecter dump_active .. 0 homeless 2024-02-06T21:41:32.282 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.282+0000 7f5faeffd640 1 -- 172.21.15.120:0/1353137138 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5fb00540d0 con 0x7f5fc00a9780 2024-02-06T21:41:32.282 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.282+0000 7f5faeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:32.282 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.282+0000 7f5faeffd640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:32.467 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.466+0000 7f5fc4964640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:32.467 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.466+0000 7f5fc4964640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:32.467 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.466+0000 7f5fc4964640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:32.467 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.466+0000 7f5fc4964640 1 -- 172.21.15.120:0/1353137138 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fc00ac890 con 0x7f5fc00a9780 2024-02-06T21:41:32.556 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.555+0000 7f5faeffd640 1 -- 172.21.15.120:0/1353137138 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+12933 (secure 0 0 0) 0x7f5fb0017020 con 0x7f5fc00a9780 2024-02-06T21:41:32.556 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.555+0000 7f5faeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:32.556 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.555+0000 7f5faeffd640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:32.557 INFO:teuthology.misc.health.smithi120.stdout: 2024-02-06T21:41:32.557 INFO:teuthology.misc.health.smithi120.stdout:{"epoch":20,"fsid":"4e4791bf-0a02-4478-9517-2d2e893f4f7c","created":"2024-02-06T21:40:31.800075+0000","modified":"2024-02-06T21:41:31.886554+0000","last_up_change":"2024-02-06T21:41:29.190309+0000","last_in_change":"2024-02-06T21:40:37.354288+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-06T21:41:20.093970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4bd0f591-2cf1-42da-8ac1-dce3188580c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6817","nonce":2062960776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6819","nonce":2062960776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6823","nonce":2062960776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6821","nonce":2062960776}]},"public_addr":"172.21.15.120:6817/2062960776","cluster_addr":"172.21.15.120:6819/2062960776","heartbeat_back_addr":"172.21.15.120:6823/2062960776","heartbeat_front_addr":"172.21.15.120:6821/2062960776","state":["exists","up"]},{"osd":1,"uuid":"81dc801c-3768-4c2c-9adb-6c8300a341f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6809","nonce":3348315891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6811","nonce":3348315891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6815","nonce":3348315891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6813","nonce":3348315891}]},"public_addr":"172.21.15.120:6809/3348315891","cluster_addr":"172.21.15.120:6811/3348315891","heartbeat_back_addr":"172.21.15.120:6815/3348315891","heartbeat_front_addr":"172.21.15.120:6813/3348315891","state":["exists","up"]},{"osd":2,"uuid":"9289721b-c13f-4a17-be68-3698090f69af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6825","nonce":3229495265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6827","nonce":3229495265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6831","nonce":3229495265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6829","nonce":3229495265}]},"public_addr":"172.21.15.120:6825/3229495265","cluster_addr":"172.21.15.120:6827/3229495265","heartbeat_back_addr":"172.21.15.120:6831/3229495265","heartbeat_front_addr":"172.21.15.120:6829/3229495265","state":["exists","up"]},{"osd":3,"uuid":"53534db0-2d4d-44ed-90e8-d67ea878b51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6801","nonce":3324566974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6803","nonce":3324566974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6807","nonce":3324566974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6805","nonce":3324566974}]},"public_addr":"172.21.15.120:6801/3324566974","cluster_addr":"172.21.15.120:6803/3324566974","heartbeat_back_addr":"172.21.15.120:6807/3324566974","heartbeat_front_addr":"172.21.15.120:6805/3324566974","state":["exists","up"]},{"osd":4,"uuid":"e4205ab5-787c-44e0-9b96-273b8c9cf1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6803","nonce":2191547918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6805","nonce":2191547918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6809","nonce":2191547918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6807","nonce":2191547918}]},"public_addr":"172.21.15.135:6803/2191547918","cluster_addr":"172.21.15.135:6805/2191547918","heartbeat_back_addr":"172.21.15.135:6809/2191547918","heartbeat_front_addr":"172.21.15.135:6807/2191547918","state":["exists","up"]},{"osd":5,"uuid":"43b600bc-3afe-44bd-bd3f-690106e7e0ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6819","nonce":563193066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6821","nonce":563193066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6826","nonce":563193066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6823","nonce":563193066}]},"public_addr":"172.21.15.135:6819/563193066","cluster_addr":"172.21.15.135:6821/563193066","heartbeat_back_addr":"172.21.15.135:6826/563193066","heartbeat_front_addr":"172.21.15.135:6823/563193066","state":["exists","up"]},{"osd":6,"uuid":"1d6b9f96-1f9d-44f2-83b0-93f5c342fafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6811","nonce":1080459988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6813","nonce":1080459988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6817","nonce":1080459988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6815","nonce":1080459988}]},"public_addr":"172.21.15.135:6811/1080459988","cluster_addr":"172.21.15.135:6813/1080459988","heartbeat_back_addr":"172.21.15.135:6817/1080459988","heartbeat_front_addr":"172.21.15.135:6815/1080459988","state":["exists","up"]},{"osd":7,"uuid":"ecae00f5-8e29-47dd-90d5-e51503bc9dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6825","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6827","nonce":3740758337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6829","nonce":3740758337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6833","nonce":3740758337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6831","nonce":3740758337}]},"public_addr":"172.21.15.135:6827/3740758337","cluster_addr":"172.21.15.135:6829/3740758337","heartbeat_back_addr":"172.21.15.135:6833/3740758337","heartbeat_front_addr":"172.21.15.135:6831/3740758337","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.344503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:17.073640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.245112+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.202999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:15.825107+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:15.918704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.203923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:15.753628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-06T21:41:32.559 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.557+0000 7f5fc4964640 10 librados: watch_flush enter 2024-02-06T21:41:32.559 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.557+0000 7f5fc4964640 10 librados: watch_flush exit 2024-02-06T21:41:32.559 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.557+0000 7f5fc4964640 20 client.4249.objecter shutdown clearing up homeless session... 2024-02-06T21:41:32.559 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.557+0000 7f5fc4964640 10 client.4249.objecter successfully canceled tick 2024-02-06T21:41:32.559 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.558+0000 7f5fc4964640 1 -- 172.21.15.120:0/1353137138 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f5fa006d110 msgr2=0x7f5fa006f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:32.559 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.558+0000 7f5fc4964640 1 --2- 172.21.15.120:0/1353137138 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f5fa006d110 0x7f5fa006f5d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0145770 tx=0x7f5fb4007930 comp rx=0 tx=0).stop 2024-02-06T21:41:32.559 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.558+0000 7f5fc4964640 10 monclient: shutdown 2024-02-06T21:41:32.559 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.558+0000 7f5fc4964640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:32.560 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.558+0000 7f5fc4964640 1 -- 172.21.15.120:0/1353137138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fc00a9780 msgr2=0x7f5fc01464a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:32.560 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.558+0000 7f5fc4964640 1 --2- 172.21.15.120:0/1353137138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fc00a9780 0x7f5fc01464a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000c410 tx=0x7f5fb000c8d0 comp rx=0 tx=0).stop 2024-02-06T21:41:32.560 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.558+0000 7f5fc4964640 1 -- 172.21.15.120:0/1353137138 shutdown_connections 2024-02-06T21:41:32.560 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.558+0000 7f5fc4964640 1 --2- 172.21.15.120:0/1353137138 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f5fa006d110 0x7f5fa006f5d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:32.560 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.558+0000 7f5fc4964640 1 --2- 172.21.15.120:0/1353137138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fc013fe70 0x7f5fc0138360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:32.560 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.558+0000 7f5fc4964640 1 --2- 172.21.15.120:0/1353137138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fc00b5060 0x7f5fc00abd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:32.560 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.558+0000 7f5fc4964640 1 --2- 172.21.15.120:0/1353137138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fc00a9780 0x7f5fc01464a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:32.560 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.558+0000 7f5fc4964640 1 -- 172.21.15.120:0/1353137138 >> 172.21.15.120:0/1353137138 conn(0x7f5fc00724d0 msgr2=0x7f5fc013f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:32.560 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.558+0000 7f5fc4964640 1 -- 172.21.15.120:0/1353137138 shutdown_connections 2024-02-06T21:41:32.560 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.558+0000 7f5fc4964640 1 -- 172.21.15.120:0/1353137138 wait complete. 2024-02-06T21:41:32.560 INFO:teuthology.misc.health.smithi120.stderr:2024-02-06T21:41:32.558+0000 7f5fc4964640 1 librados: shutdown 2024-02-06T21:41:32.568 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-02-06T21:41:32.568 INFO:tasks.ceph:Creating RBD pool 2024-02-06T21:41:32.569 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd pool create rbd 8 2024-02-06T21:41:32.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.669+0000 7f959187b640 10 monclient: get_monmap_and_config 2024-02-06T21:41:32.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.669+0000 7f959187b640 10 monclient: build_initial_monmap 2024-02-06T21:41:32.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.669+0000 7f959187b640 10 monclient: monmap: 2024-02-06T21:41:32.670 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:32.670 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:32.670 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:32.669707+0000 2024-02-06T21:41:32.670 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:32.669707+0000 2024-02-06T21:41:32.670 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:32.670 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:32.670 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:32.670 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:32.670 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:32.670 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:32.671 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:32.671 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:32.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 1 Processor -- start 2024-02-06T21:41:32.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 1 -- start start 2024-02-06T21:41:32.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 10 monclient: init 2024-02-06T21:41:32.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:32.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-02-06T21:41:32.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f958c0a9c80 0x7f958c0aa060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:32.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 10 monclient(hunting): picked mon.noname-a con 0x7f958c0a9c80 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:32.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f958c142bb0 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:32.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f958c13ff20 0x7f958c142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:32.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 10 monclient(hunting): picked mon.noname-b con 0x7f958c13ff20 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:32.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f958c0aa5a0 con 0x7f958c142bb0 2024-02-06T21:41:32.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:32.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f958c143510 con 0x7f958c0a9c80 2024-02-06T21:41:32.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:32.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958c142720 con 0x7f958c13ff20 2024-02-06T21:41:32.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:32.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.670+0000 7f959187b640 10 monclient(hunting): authenticate will time out at 1323.007269s 2024-02-06T21:41:32.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.671+0000 7f958affd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f958c0a9c80 0x7f958c0aa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:32.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.671+0000 7f958b7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f958c13ff20 0x7f958c142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:32.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.671+0000 7f958affd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f958c0a9c80 0x7f958c0aa060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60004/0 (socket says 172.21.15.120:60004) 2024-02-06T21:41:32.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.671+0000 7f958affd640 1 -- 172.21.15.120:0/2491591227 learned_addr learned my addr 172.21.15.120:0/2491591227 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:32.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.671+0000 7f958affd640 10 monclient(hunting): get_auth_request con 0x7f958c0a9c80 auth_method 0 2024-02-06T21:41:32.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.671+0000 7f958affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:32.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.671+0000 7f958affd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:32.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.671+0000 7f958affd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:32.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.671+0000 7f958b7fe640 10 monclient(hunting): get_auth_request con 0x7f958c13ff20 auth_method 0 2024-02-06T21:41:32.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.671+0000 7f958b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:32.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.671+0000 7f958b7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:32.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.671+0000 7f958b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:32.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.672+0000 7f958b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:32.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.672+0000 7f958b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:32.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.672+0000 7f958b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:32.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.672+0000 7f958affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:32.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.672+0000 7f958affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:32.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.672+0000 7f958affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:32.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.674+0000 7f958b7fe640 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2024-02-06T21:41:32.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.674+0000 7f958b7fe640 1 -- 172.21.15.120:0/2491591227 >> v2:172.21.15.120:3300/0 conn(0x7f958c0a9c80 msgr2=0x7f958c0aa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:32.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.674+0000 7f958b7fe640 1 --2- 172.21.15.120:0/2491591227 >> v2:172.21.15.120:3300/0 conn(0x7f958c0a9c80 0x7f958c0aa060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:32.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.674+0000 7f958b7fe640 1 -- 172.21.15.120:0/2491591227 >> v1:172.21.15.120:6789/0 conn(0x7f958c142bb0 legacy=0x7f958c13f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:32.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.674+0000 7f958b7fe640 10 monclient: _finish_hunting 0 2024-02-06T21:41:32.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.674+0000 7f958b7fe640 1 monclient: found mon.noname-b 2024-02-06T21:41:32.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.674+0000 7f958b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:32.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.674+0000 7f958b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:32.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.674+0000 7f958b7fe640 1 -- 172.21.15.120:0/2491591227 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958c0ab120 con 0x7f958c13ff20 2024-02-06T21:41:32.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.675+0000 7f958b7fe640 1 --2- 172.21.15.120:0/2491591227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f958c13ff20 0x7f958c142fd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9584009d90 tx=0x7f958402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cbc19090f2f7860 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:32.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f9589ffb640 1 -- 172.21.15.120:0/2491591227 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958402cc50 con 0x7f958c13ff20 2024-02-06T21:41:32.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f9589ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:32.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f9589ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:32.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f9589ffb640 10 monclient: dump: 2024-02-06T21:41:32.680 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:32.680 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:32.680 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:32.680 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:32.681 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:32.681 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:32.681 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:32.681 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:32.681 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:32.681 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:32.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f9589ffb640 10 monclient: _finish_auth 0 2024-02-06T21:41:32.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f9589ffb640 10 monclient: _check_auth_tickets 2024-02-06T21:41:32.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f9589ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:32.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f9589ffb640 1 -- 172.21.15.120:0/2491591227 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958402cdb0 con 0x7f958c13ff20 2024-02-06T21:41:32.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f9589ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:32.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f9589ffb640 1 -- 172.21.15.120:0/2491591227 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9584034660 con 0x7f958c13ff20 2024-02-06T21:41:32.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f959187b640 5 monclient: authenticate success, global_id 4232 2024-02-06T21:41:32.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f959187b640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:32.682 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f959187b640 4 set_mon_vals no callback set 2024-02-06T21:41:32.682 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f9589ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:32.682 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f9589ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:32.682 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.679+0000 7f9589ffb640 10 monclient: dump: 2024-02-06T21:41:32.682 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:32.682 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:32.682 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:32.682 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:32.682 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:32.682 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:32.682 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:32.682 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:32.682 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:32.682 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:32.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.680+0000 7f959187b640 10 monclient: shutdown 2024-02-06T21:41:32.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.680+0000 7f959187b640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:32.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.680+0000 7f959187b640 1 -- 172.21.15.120:0/2491591227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f958c13ff20 msgr2=0x7f958c142fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:32.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.680+0000 7f959187b640 1 --2- 172.21.15.120:0/2491591227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f958c13ff20 0x7f958c142fd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9584009d90 tx=0x7f958402bea0 comp rx=0 tx=0).stop 2024-02-06T21:41:32.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.680+0000 7f959187b640 1 -- 172.21.15.120:0/2491591227 shutdown_connections 2024-02-06T21:41:32.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.680+0000 7f959187b640 1 --2- 172.21.15.120:0/2491591227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f958c13ff20 0x7f958c142fd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:32.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.680+0000 7f959187b640 1 --2- 172.21.15.120:0/2491591227 >> v2:172.21.15.120:3300/0 conn(0x7f958c0a9c80 0x7f958c0aa060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:32.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.680+0000 7f959187b640 1 -- 172.21.15.120:0/2491591227 >> 172.21.15.120:0/2491591227 conn(0x7f958c0b4ba0 msgr2=0x7f958c0b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:32.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.680+0000 7f959187b640 1 -- 172.21.15.120:0/2491591227 shutdown_connections 2024-02-06T21:41:32.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.680+0000 7f959187b640 1 -- 172.21.15.120:0/2491591227 wait complete. 2024-02-06T21:41:32.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.681+0000 7f959187b640 10 monclient: build_initial_monmap 2024-02-06T21:41:32.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.681+0000 7f959187b640 10 monclient: monmap: 2024-02-06T21:41:32.686 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:32.687 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.681+0000 7f959187b640 1 librados: starting msgr at 2024-02-06T21:41:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.681+0000 7f959187b640 1 librados: starting objecter 2024-02-06T21:41:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.681+0000 7f959187b640 1 Processor -- start 2024-02-06T21:41:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 1 -- start start 2024-02-06T21:41:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 1 librados: setting wanted keys 2024-02-06T21:41:32.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 1 librados: calling monclient init 2024-02-06T21:41:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 10 monclient: init 2024-02-06T21:41:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T21:41:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f958c0a9c80 0x7f958c138660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 10 monclient(hunting): picked mon.noname-c con 0x7f958c0a9c80 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f958c13ff20 0x7f958c1db070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 10 monclient(hunting): picked mon.noname-b con 0x7f958c13ff20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c142bb0 0x7f958c1db5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 10 monclient(hunting): picked mon.noname-a con 0x7f958c142bb0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c1dac20 con 0x7f958c142bb0 2024-02-06T21:41:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c145840 con 0x7f958c13ff20 2024-02-06T21:41:32.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958c139910 con 0x7f958c0a9c80 2024-02-06T21:41:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.682+0000 7f959187b640 10 monclient(hunting): authenticate will time out at 1323.019257s 2024-02-06T21:41:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958affd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f958c0a9c80 0x7f958c138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958a7fc640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f958c13ff20 0x7f958c1db070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c142bb0 0x7f958c1db5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958a7fc640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f958c13ff20 0x7f958c1db070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38458/0 (socket says 172.21.15.120:38458) 2024-02-06T21:41:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958a7fc640 1 -- 172.21.15.120:0/2742720301 learned_addr learned my addr 172.21.15.120:0/2742720301 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958a7fc640 10 monclient(hunting): get_auth_request con 0x7f958c13ff20 auth_method 0 2024-02-06T21:41:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958a7fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958affd640 10 monclient(hunting): get_auth_request con 0x7f958c0a9c80 auth_method 0 2024-02-06T21:41:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:32.689 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958affd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:32.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958affd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:32.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958b7fe640 10 monclient(hunting): get_auth_request con 0x7f958c142bb0 auth_method 0 2024-02-06T21:41:32.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:32.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958b7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:32.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.683+0000 7f958b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:32.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.684+0000 7f958a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:32.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.684+0000 7f958a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:32.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.684+0000 7f958a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:32.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.684+0000 7f958affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:32.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.684+0000 7f958affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:32.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.684+0000 7f958affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:32.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.684+0000 7f958b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:32.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.684+0000 7f958b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:32.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.684+0000 7f958b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:32.691 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.686+0000 7f958a7fc640 10 monclient(hunting): handle_auth_done global_id 4252 payload 824 2024-02-06T21:41:32.691 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.687+0000 7f958a7fc640 1 -- 172.21.15.120:0/2742720301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f958c0a9c80 msgr2=0x7f958c138660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:32.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.687+0000 7f958a7fc640 1 --2- 172.21.15.120:0/2742720301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f958c0a9c80 0x7f958c138660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:32.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.687+0000 7f958a7fc640 1 -- 172.21.15.120:0/2742720301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c142bb0 msgr2=0x7f958c1db5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:32.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.687+0000 7f958a7fc640 1 --2- 172.21.15.120:0/2742720301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c142bb0 0x7f958c1db5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:32.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.687+0000 7f958a7fc640 10 monclient: _finish_hunting 0 2024-02-06T21:41:32.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.687+0000 7f958a7fc640 1 monclient: found mon.noname-b 2024-02-06T21:41:32.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.687+0000 7f958a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:32.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.687+0000 7f958a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:32.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.687+0000 7f958a7fc640 1 -- 172.21.15.120:0/2742720301 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9584009a70 con 0x7f958c13ff20 2024-02-06T21:41:32.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.689+0000 7f958a7fc640 1 --2- 172.21.15.120:0/2742720301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f958c13ff20 0x7f958c1db070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f957c00c3f0 tx=0x7f957c00c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:32.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.689+0000 7f9573fff640 10 client.?.objecter ms_handle_connect 0x7f958c13ff20 2024-02-06T21:41:32.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.689+0000 7f9573fff640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:32.695 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f9573fff640 1 -- 172.21.15.120:0/2742720301 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c0052d0 con 0x7f958c13ff20 2024-02-06T21:41:32.695 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f9573fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:32.695 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f9573fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:32.695 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f9573fff640 10 monclient: dump: 2024-02-06T21:41:32.695 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:32.695 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:32.695 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:32.695 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:32.695 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:32.696 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:32.696 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:32.696 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:32.696 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:32.696 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:32.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f9573fff640 10 monclient: _finish_auth 0 2024-02-06T21:41:32.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f9573fff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:32.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f9573fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:32.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f9573fff640 1 -- 172.21.15.120:0/2742720301 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957c005450 con 0x7f958c13ff20 2024-02-06T21:41:32.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f9573fff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:32.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f9573fff640 1 -- 172.21.15.120:0/2742720301 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c007260 con 0x7f958c13ff20 2024-02-06T21:41:32.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f959187b640 5 monclient: authenticate success, global_id 4252 2024-02-06T21:41:32.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f959187b640 10 monclient: _renew_subs 2024-02-06T21:41:32.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f959187b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:32.697 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f959187b640 1 -- 172.21.15.120:0/2742720301 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958c139be0 con 0x7f958c13ff20 2024-02-06T21:41:32.697 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f9573fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:32.697 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f9573fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:32.697 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f9573fff640 10 monclient: dump: 2024-02-06T21:41:32.697 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:32.697 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:32.697 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:32.697 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:32.697 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:32.697 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:32.697 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:32.697 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:32.697 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:32.697 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:32.697 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f95897fa640 4 set_mon_vals no callback set 2024-02-06T21:41:32.698 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f959187b640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:32.698 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f959187b640 10 monclient: _renew_subs 2024-02-06T21:41:32.698 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f959187b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:32.698 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f959187b640 1 -- 172.21.15.120:0/2742720301 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958c138ba0 con 0x7f958c13ff20 2024-02-06T21:41:32.698 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.694+0000 7f959187b640 1 librados: init done 2024-02-06T21:41:32.698 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.695+0000 7f959187b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:32.698 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.695+0000 7f959187b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:32.698 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.695+0000 7f959187b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:32.698 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.695+0000 7f959187b640 1 -- 172.21.15.120:0/2742720301 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f958c04ef90 con 0x7f958c13ff20 2024-02-06T21:41:32.715 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.715+0000 7f9573fff640 1 -- 172.21.15.120:0/2742720301 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 4) v1 ==== 89732+0+0 (secure 0 0 0) 0x7f957c0073c0 con 0x7f958c13ff20 2024-02-06T21:41:32.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.715+0000 7f9573fff640 1 --2- 172.21.15.120:0/2742720301 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f957806d160 0x7f957806f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:32.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.715+0000 7f958affd640 1 --2- 172.21.15.120:0/2742720301 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f957806d160 0x7f957806f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:32.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.716+0000 7f958affd640 10 monclient: get_auth_request con 0x7f957806d160 auth_method 0 2024-02-06T21:41:32.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.716+0000 7f958affd640 1 --2- 172.21.15.120:0/2742720301 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f957806d160 0x7f957806f620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9580009bd0 tx=0x7f9580009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:32.717 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.716+0000 7f9573fff640 10 client.4252.objecter ms_handle_connect 0x7f957806d160 2024-02-06T21:41:32.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.939+0000 7f9573fff640 1 -- 172.21.15.120:0/2742720301 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f957c058330 con 0x7f958c13ff20 2024-02-06T21:41:32.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.939+0000 7f9573fff640 10 client.4252.objecter ms_dispatch 0x7f958c000b90 osd_map(20..20 src has 1..20) v4 2024-02-06T21:41:32.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.939+0000 7f9573fff640 3 client.4252.objecter handle_osd_map got epochs [20,20] > 0 2024-02-06T21:41:32.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.939+0000 7f9573fff640 3 client.4252.objecter handle_osd_map decoding full epoch 20 2024-02-06T21:41:32.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.939+0000 7f9573fff640 20 client.4252.objecter dump_active .. 0 homeless 2024-02-06T21:41:32.952 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.951+0000 7f9573fff640 1 -- 172.21.15.120:0/2742720301 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f957c0545d0 con 0x7f958c13ff20 2024-02-06T21:41:32.952 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.951+0000 7f9573fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:32.952 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:32.951+0000 7f9573fff640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:33.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:33.171+0000 7f959187b640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-02-06T21:41:33.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:33.171+0000 7f959187b640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-02-06T21:41:33.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:33.171+0000 7f959187b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:33.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:33.171+0000 7f959187b640 1 -- 172.21.15.120:0/2742720301 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f958c0aa5a0 con 0x7f958c13ff20 2024-02-06T21:41:34.278 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.278+0000 7f9573fff640 1 -- 172.21.15.120:0/2742720301 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v21) v1 ==== 109+0+0 (secure 0 0 0) 0x7f958c0aa5a0 con 0x7f958c13ff20 2024-02-06T21:41:34.278 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.278+0000 7f9573fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-02-06T21:41:34.279 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.278+0000 7f9573fff640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-02-06T21:41:34.279 INFO:teuthology.orchestra.run.smithi120.stderr:pool 'rbd' created 2024-02-06T21:41:34.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.280+0000 7f959187b640 10 librados: watch_flush enter 2024-02-06T21:41:34.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.280+0000 7f959187b640 10 librados: watch_flush exit 2024-02-06T21:41:34.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.280+0000 7f959187b640 20 client.4252.objecter shutdown clearing up homeless session... 2024-02-06T21:41:34.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.280+0000 7f959187b640 10 client.4252.objecter successfully canceled tick 2024-02-06T21:41:34.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.281+0000 7f959187b640 1 -- 172.21.15.120:0/2742720301 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f957806d160 msgr2=0x7f957806f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:34.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.281+0000 7f959187b640 1 --2- 172.21.15.120:0/2742720301 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f957806d160 0x7f957806f620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9580009bd0 tx=0x7f9580009240 comp rx=0 tx=0).stop 2024-02-06T21:41:34.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.281+0000 7f959187b640 10 monclient: shutdown 2024-02-06T21:41:34.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.281+0000 7f959187b640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:34.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.281+0000 7f959187b640 1 -- 172.21.15.120:0/2742720301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f958c13ff20 msgr2=0x7f958c1db070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:34.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.281+0000 7f959187b640 1 --2- 172.21.15.120:0/2742720301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f958c13ff20 0x7f958c1db070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f957c00c3f0 tx=0x7f957c00c8b0 comp rx=0 tx=0).stop 2024-02-06T21:41:34.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.281+0000 7f959187b640 1 -- 172.21.15.120:0/2742720301 shutdown_connections 2024-02-06T21:41:34.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.281+0000 7f959187b640 1 --2- 172.21.15.120:0/2742720301 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f957806d160 0x7f957806f620 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:34.283 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.281+0000 7f959187b640 1 --2- 172.21.15.120:0/2742720301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f958c13ff20 0x7f958c1db070 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:34.283 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.281+0000 7f959187b640 1 --2- 172.21.15.120:0/2742720301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c142bb0 0x7f958c1db5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:34.283 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.281+0000 7f959187b640 1 --2- 172.21.15.120:0/2742720301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f958c0a9c80 0x7f958c138660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:34.283 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.281+0000 7f959187b640 1 -- 172.21.15.120:0/2742720301 >> 172.21.15.120:0/2742720301 conn(0x7f958c0b4ba0 msgr2=0x7f958c0b37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:34.283 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.281+0000 7f959187b640 1 -- 172.21.15.120:0/2742720301 shutdown_connections 2024-02-06T21:41:34.283 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.281+0000 7f959187b640 1 -- 172.21.15.120:0/2742720301 wait complete. 2024-02-06T21:41:34.283 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.281+0000 7f959187b640 1 librados: shutdown 2024-02-06T21:41:34.296 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-02-06T21:41:34.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.402+0000 7fbf53723640 10 monclient: get_monmap_and_config 2024-02-06T21:41:34.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.402+0000 7fbf53723640 10 monclient: build_initial_monmap 2024-02-06T21:41:34.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.402+0000 7fbf53723640 10 monclient: monmap: 2024-02-06T21:41:34.403 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:34.403 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:34.403 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:34.402372+0000 2024-02-06T21:41:34.403 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:34.402372+0000 2024-02-06T21:41:34.403 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:34.403 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:34.403 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:34.403 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:34.403 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:34.403 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:34.403 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:34.403 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:34.405 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.402+0000 7fbf53723640 1 Processor -- start 2024-02-06T21:41:34.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.402+0000 7fbf53723640 1 -- start start 2024-02-06T21:41:34.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.402+0000 7fbf53723640 10 monclient: init 2024-02-06T21:41:34.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.403+0000 7fbf53723640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:34.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.403+0000 7fbf53723640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-02-06T21:41:34.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.403+0000 7fbf53723640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbf4c0b5230 0x7fbf4c0a9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:34.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.403+0000 7fbf53723640 10 monclient(hunting): picked mon.noname-c con 0x7fbf4c0b5230 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:34.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.403+0000 7fbf53723640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbf4c13f960 0x7fbf4c13fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:34.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.403+0000 7fbf53723640 10 monclient(hunting): picked mon.noname-a con 0x7fbf4c13f960 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:34.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.403+0000 7fbf53723640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf4c1425f0 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:34.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.403+0000 7fbf53723640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf4c0a9d90 con 0x7fbf4c1425f0 2024-02-06T21:41:34.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.403+0000 7fbf53723640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:34.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.403+0000 7fbf53723640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4c0b5610 con 0x7fbf4c13f960 2024-02-06T21:41:34.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.403+0000 7fbf53723640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:34.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.403+0000 7fbf53723640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4c0ab3e0 con 0x7fbf4c0b5230 2024-02-06T21:41:34.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.403+0000 7fbf53723640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:34.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.403+0000 7fbf53723640 10 monclient(hunting): authenticate will time out at 1324.740125s 2024-02-06T21:41:34.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.404+0000 7fbf514bf640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbf4c0b5230 0x7fbf4c0a9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:34.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.404+0000 7fbf50cbe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbf4c13f960 0x7fbf4c13fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:34.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.404+0000 7fbf514bf640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbf4c0b5230 0x7fbf4c0a9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38460/0 (socket says 172.21.15.120:38460) 2024-02-06T21:41:34.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.404+0000 7fbf514bf640 1 -- 172.21.15.120:0/1017056030 learned_addr learned my addr 172.21.15.120:0/1017056030 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:34.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.404+0000 7fbf514bf640 10 monclient(hunting): get_auth_request con 0x7fbf4c0b5230 auth_method 0 2024-02-06T21:41:34.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.404+0000 7fbf514bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:34.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.404+0000 7fbf514bf640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:34.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.404+0000 7fbf514bf640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:34.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.404+0000 7fbf50cbe640 10 monclient(hunting): get_auth_request con 0x7fbf4c13f960 auth_method 0 2024-02-06T21:41:34.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.404+0000 7fbf50cbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:34.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.404+0000 7fbf50cbe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:34.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.404+0000 7fbf50cbe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:34.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.405+0000 7fbf514bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:34.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.405+0000 7fbf514bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:34.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.405+0000 7fbf514bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:34.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.405+0000 7fbf50cbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:34.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.405+0000 7fbf50cbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:34.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.405+0000 7fbf50cbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:34.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.406+0000 7fbf514bf640 10 monclient(hunting): handle_auth_done global_id 4261 payload 274 2024-02-06T21:41:34.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.406+0000 7fbf514bf640 1 -- 172.21.15.120:0/1017056030 >> v2:172.21.15.120:3300/0 conn(0x7fbf4c13f960 msgr2=0x7fbf4c13fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:34.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.406+0000 7fbf514bf640 1 --2- 172.21.15.120:0/1017056030 >> v2:172.21.15.120:3300/0 conn(0x7fbf4c13f960 0x7fbf4c13fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:34.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.406+0000 7fbf514bf640 1 -- 172.21.15.120:0/1017056030 >> v1:172.21.15.120:6789/0 conn(0x7fbf4c1425f0 legacy=0x7fbf4c142a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:34.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.406+0000 7fbf514bf640 10 monclient: _finish_hunting 0 2024-02-06T21:41:34.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.406+0000 7fbf514bf640 1 monclient: found mon.noname-c 2024-02-06T21:41:34.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.406+0000 7fbf514bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:34.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.406+0000 7fbf514bf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T21:41:34.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.406+0000 7fbf514bf640 1 -- 172.21.15.120:0/1017056030 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c0ab6c0 con 0x7fbf4c0b5230 2024-02-06T21:41:34.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.408+0000 7fbf514bf640 1 --2- 172.21.15.120:0/1017056030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf4c0b5230 0x7fbf4c0a9850 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0b9040 tx=0x7fbf4802b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3d54e1b42a3be2b server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:34.413 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf43fff640 1 -- 172.21.15.120:0/1017056030 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf480317e0 con 0x7fbf4c0b5230 2024-02-06T21:41:34.413 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:34.413 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf43fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:34.413 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf43fff640 10 monclient: dump: 2024-02-06T21:41:34.413 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:34.413 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:34.413 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:34.413 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf43fff640 10 monclient: _finish_auth 0 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf43fff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf43fff640 1 -- 172.21.15.120:0/1017056030 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf48031940 con 0x7fbf4c0b5230 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf53723640 5 monclient: authenticate success, global_id 4261 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf53723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf43fff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf53723640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf43fff640 1 -- 172.21.15.120:0/1017056030 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48031cd0 con 0x7fbf4c0b5230 2024-02-06T21:41:34.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf53723640 4 set_mon_vals no callback set 2024-02-06T21:41:34.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:34.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:34.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.412+0000 7fbf43fff640 10 monclient: dump: 2024-02-06T21:41:34.415 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:34.415 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:34.415 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:34.415 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:34.415 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:34.415 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:34.415 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:34.415 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:34.415 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:34.415 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:34.419 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.413+0000 7fbf53723640 10 monclient: shutdown 2024-02-06T21:41:34.419 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.413+0000 7fbf53723640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:34.419 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.413+0000 7fbf53723640 1 -- 172.21.15.120:0/1017056030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf4c0b5230 msgr2=0x7fbf4c0a9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:34.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.413+0000 7fbf53723640 1 --2- 172.21.15.120:0/1017056030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf4c0b5230 0x7fbf4c0a9850 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0b9040 tx=0x7fbf4802b880 comp rx=0 tx=0).stop 2024-02-06T21:41:34.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.413+0000 7fbf53723640 1 -- 172.21.15.120:0/1017056030 shutdown_connections 2024-02-06T21:41:34.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.413+0000 7fbf53723640 1 --2- 172.21.15.120:0/1017056030 >> v2:172.21.15.120:3300/0 conn(0x7fbf4c13f960 0x7fbf4c13fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:34.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.413+0000 7fbf53723640 1 --2- 172.21.15.120:0/1017056030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf4c0b5230 0x7fbf4c0a9850 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:34.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.413+0000 7fbf53723640 1 -- 172.21.15.120:0/1017056030 >> 172.21.15.120:0/1017056030 conn(0x7fbf4c06d1c0 msgr2=0x7fbf4c06d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:34.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.413+0000 7fbf53723640 1 -- 172.21.15.120:0/1017056030 shutdown_connections 2024-02-06T21:41:34.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.413+0000 7fbf53723640 1 -- 172.21.15.120:0/1017056030 wait complete. 2024-02-06T21:41:34.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.414+0000 7fbf53723640 10 monclient: build_initial_monmap 2024-02-06T21:41:34.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.414+0000 7fbf53723640 10 monclient: monmap: 2024-02-06T21:41:34.420 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:34.420 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:34.420 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:34.420 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:34.420 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:34.421 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:34.421 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:34.421 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:34.421 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:34.421 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:34.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.414+0000 7fbf53723640 1 librados: starting msgr at 2024-02-06T21:41:34.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.414+0000 7fbf53723640 1 librados: starting objecter 2024-02-06T21:41:34.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.414+0000 7fbf53723640 1 Processor -- start 2024-02-06T21:41:34.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.414+0000 7fbf53723640 1 -- start start 2024-02-06T21:41:34.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.414+0000 7fbf53723640 1 librados: setting wanted keys 2024-02-06T21:41:34.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.414+0000 7fbf53723640 1 librados: calling monclient init 2024-02-06T21:41:34.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.414+0000 7fbf53723640 10 monclient: init 2024-02-06T21:41:34.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:34.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T21:41:34.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf4c0b5230 0x7fbf4c138680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:34.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 10 monclient(hunting): picked mon.noname-b con 0x7fbf4c0b5230 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:34.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf4c13f960 0x7fbf4c13a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:34.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 10 monclient(hunting): picked mon.noname-a con 0x7fbf4c13f960 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:34.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf4c1425f0 0x7fbf4c137450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:34.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 10 monclient(hunting): picked mon.noname-c con 0x7fbf4c1425f0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:34.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:34.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c13c370 con 0x7fbf4c13f960 2024-02-06T21:41:34.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:34.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c13c570 con 0x7fbf4c0b5230 2024-02-06T21:41:34.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:34.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c13aae0 con 0x7fbf4c1425f0 2024-02-06T21:41:34.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf53723640 10 monclient(hunting): authenticate will time out at 1324.751805s 2024-02-06T21:41:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf51cc0640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf4c1425f0 0x7fbf4c137450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf514bf640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf4c0b5230 0x7fbf4c138680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.415+0000 7fbf50cbe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf4c13f960 0x7fbf4c13a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf514bf640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf4c0b5230 0x7fbf4c138680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38472/0 (socket says 172.21.15.120:38472) 2024-02-06T21:41:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf514bf640 1 -- 172.21.15.120:0/2086785919 learned_addr learned my addr 172.21.15.120:0/2086785919 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf514bf640 10 monclient(hunting): get_auth_request con 0x7fbf4c0b5230 auth_method 0 2024-02-06T21:41:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf514bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf514bf640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf514bf640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf51cc0640 10 monclient(hunting): get_auth_request con 0x7fbf4c1425f0 auth_method 0 2024-02-06T21:41:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf51cc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf51cc0640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf51cc0640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:34.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf50cbe640 10 monclient(hunting): get_auth_request con 0x7fbf4c13f960 auth_method 0 2024-02-06T21:41:34.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf50cbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:34.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf50cbe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:34.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf50cbe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:34.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf514bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:34.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf514bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:34.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf514bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:34.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf50cbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:34.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf50cbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:34.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf50cbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:34.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf51cc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:34.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf51cc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:34.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.416+0000 7fbf51cc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:34.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.419+0000 7fbf514bf640 10 monclient(hunting): handle_auth_done global_id 4264 payload 824 2024-02-06T21:41:34.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.419+0000 7fbf514bf640 1 -- 172.21.15.120:0/2086785919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf4c1425f0 msgr2=0x7fbf4c137450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:34.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.419+0000 7fbf514bf640 1 --2- 172.21.15.120:0/2086785919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf4c1425f0 0x7fbf4c137450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:34.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.419+0000 7fbf514bf640 1 -- 172.21.15.120:0/2086785919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf4c13f960 msgr2=0x7fbf4c13a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:34.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.419+0000 7fbf514bf640 1 --2- 172.21.15.120:0/2086785919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf4c13f960 0x7fbf4c13a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:34.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.419+0000 7fbf514bf640 10 monclient: _finish_hunting 0 2024-02-06T21:41:34.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.419+0000 7fbf514bf640 1 monclient: found mon.noname-b 2024-02-06T21:41:34.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.419+0000 7fbf514bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:34.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.419+0000 7fbf514bf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:34.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.419+0000 7fbf514bf640 1 -- 172.21.15.120:0/2086785919 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf480094d0 con 0x7fbf4c0b5230 2024-02-06T21:41:34.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.423+0000 7fbf514bf640 1 --2- 172.21.15.120:0/2086785919 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf4c0b5230 0x7fbf4c138680 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbf48000c00 tx=0x7fbf480046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:34.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.423+0000 7fbf41ffb640 10 client.?.objecter ms_handle_connect 0x7fbf4c0b5230 2024-02-06T21:41:34.426 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.423+0000 7fbf41ffb640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:34.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.427+0000 7fbf41ffb640 1 -- 172.21.15.120:0/2086785919 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48032880 con 0x7fbf4c0b5230 2024-02-06T21:41:34.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:34.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf41ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:34.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf41ffb640 10 monclient: dump: 2024-02-06T21:41:34.429 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:34.429 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:34.429 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:34.429 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:34.429 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:34.429 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:34.429 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:34.429 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:34.429 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:34.429 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:34.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf41ffb640 10 monclient: _finish_auth 0 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf41ffb640 10 monclient: _check_auth_tickets 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf41ffb640 1 -- 172.21.15.120:0/2086785919 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf480329e0 con 0x7fbf4c0b5230 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf41ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf41ffb640 1 -- 172.21.15.120:0/2086785919 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48032d90 con 0x7fbf4c0b5230 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf41ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf41ffb640 10 monclient: dump: 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:34.430 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:34.431 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:34.431 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:34.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf53723640 5 monclient: authenticate success, global_id 4264 2024-02-06T21:41:34.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf53723640 10 monclient: _renew_subs 2024-02-06T21:41:34.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf53723640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:34.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf53723640 1 -- 172.21.15.120:0/2086785919 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf4c13a730 con 0x7fbf4c0b5230 2024-02-06T21:41:34.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf437fe640 4 set_mon_vals no callback set 2024-02-06T21:41:34.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf53723640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:34.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf53723640 10 monclient: _renew_subs 2024-02-06T21:41:34.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf53723640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:34.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf53723640 1 -- 172.21.15.120:0/2086785919 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf4c13b000 con 0x7fbf4c0b5230 2024-02-06T21:41:34.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.428+0000 7fbf53723640 1 librados: init done 2024-02-06T21:41:34.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.429+0000 7fbf53723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:34.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.429+0000 7fbf53723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:34.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.429+0000 7fbf53723640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:34.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.429+0000 7fbf53723640 1 -- 172.21.15.120:0/2086785919 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4c04ef90 con 0x7fbf4c0b5230 2024-02-06T21:41:34.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.447+0000 7fbf41ffb640 1 -- 172.21.15.120:0/2086785919 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 4) v1 ==== 89732+0+0 (secure 0 0 0) 0x7fbf480453f0 con 0x7fbf4c0b5230 2024-02-06T21:41:34.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.447+0000 7fbf41ffb640 1 --2- 172.21.15.120:0/2086785919 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbf3006d160 0x7fbf3006f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:34.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.447+0000 7fbf50cbe640 1 --2- 172.21.15.120:0/2086785919 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbf3006d160 0x7fbf3006f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:34.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.447+0000 7fbf50cbe640 10 monclient: get_auth_request con 0x7fbf3006d160 auth_method 0 2024-02-06T21:41:34.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.448+0000 7fbf50cbe640 1 --2- 172.21.15.120:0/2086785919 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbf3006d160 0x7fbf3006f620 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c1336b0 tx=0x7fbf3c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:34.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.448+0000 7fbf41ffb640 10 client.4264.objecter ms_handle_connect 0x7fbf3006d160 2024-02-06T21:41:34.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.449+0000 7fbf41ffb640 1 -- 172.21.15.120:0/2086785919 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5199+0+0 (secure 0 0 0) 0x7fbf4807b520 con 0x7fbf4c0b5230 2024-02-06T21:41:34.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.449+0000 7fbf41ffb640 10 client.4264.objecter ms_dispatch 0x7fbf4c000b90 osd_map(21..21 src has 1..21) v4 2024-02-06T21:41:34.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.449+0000 7fbf41ffb640 3 client.4264.objecter handle_osd_map got epochs [21,21] > 0 2024-02-06T21:41:34.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.449+0000 7fbf41ffb640 3 client.4264.objecter handle_osd_map decoding full epoch 21 2024-02-06T21:41:34.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.449+0000 7fbf41ffb640 20 client.4264.objecter dump_active .. 0 homeless 2024-02-06T21:41:34.695 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.693+0000 7fbf41ffb640 1 -- 172.21.15.120:0/2086785919 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbf480777c0 con 0x7fbf4c0b5230 2024-02-06T21:41:34.695 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.693+0000 7fbf41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:34.695 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.693+0000 7fbf41ffb640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:34.920 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.918+0000 7fbf53723640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-02-06T21:41:34.920 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.919+0000 7fbf53723640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-02-06T21:41:34.920 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.919+0000 7fbf53723640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:34.921 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:34.919+0000 7fbf53723640 1 -- 172.21.15.120:0/2086785919 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbf4c140110 con 0x7fbf4c0b5230 2024-02-06T21:41:35.363 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.361+0000 7fbf41ffb640 1 -- 172.21.15.120:0/2086785919 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v22) v1 ==== 173+0+0 (secure 0 0 0) 0x7fbf48053070 con 0x7fbf4c0b5230 2024-02-06T21:41:35.363 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.361+0000 7fbf41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-02-06T21:41:35.363 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.361+0000 7fbf41ffb640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-02-06T21:41:35.363 INFO:teuthology.orchestra.run.smithi120.stderr:enabled application 'rbd' on pool 'rbd' 2024-02-06T21:41:35.368 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.364+0000 7fbf53723640 10 librados: watch_flush enter 2024-02-06T21:41:35.368 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.364+0000 7fbf53723640 10 librados: watch_flush exit 2024-02-06T21:41:35.368 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.364+0000 7fbf53723640 20 client.4264.objecter shutdown clearing up homeless session... 2024-02-06T21:41:35.368 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.364+0000 7fbf53723640 10 client.4264.objecter successfully canceled tick 2024-02-06T21:41:35.368 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.364+0000 7fbf53723640 1 -- 172.21.15.120:0/2086785919 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbf3006d160 msgr2=0x7fbf3006f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:35.368 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.365+0000 7fbf53723640 1 --2- 172.21.15.120:0/2086785919 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbf3006d160 0x7fbf3006f620 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c1336b0 tx=0x7fbf3c009190 comp rx=0 tx=0).stop 2024-02-06T21:41:35.368 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.365+0000 7fbf53723640 10 monclient: shutdown 2024-02-06T21:41:35.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.365+0000 7fbf53723640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:35.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.365+0000 7fbf53723640 1 -- 172.21.15.120:0/2086785919 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf4c0b5230 msgr2=0x7fbf4c138680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:35.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.365+0000 7fbf53723640 1 --2- 172.21.15.120:0/2086785919 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf4c0b5230 0x7fbf4c138680 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbf48000c00 tx=0x7fbf480046c0 comp rx=0 tx=0).stop 2024-02-06T21:41:35.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.365+0000 7fbf53723640 1 -- 172.21.15.120:0/2086785919 shutdown_connections 2024-02-06T21:41:35.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.365+0000 7fbf53723640 1 --2- 172.21.15.120:0/2086785919 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbf3006d160 0x7fbf3006f620 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:35.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.365+0000 7fbf53723640 1 --2- 172.21.15.120:0/2086785919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf4c1425f0 0x7fbf4c137450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:35.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.365+0000 7fbf53723640 1 --2- 172.21.15.120:0/2086785919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf4c13f960 0x7fbf4c13a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:35.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.365+0000 7fbf53723640 1 --2- 172.21.15.120:0/2086785919 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf4c0b5230 0x7fbf4c138680 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:35.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.365+0000 7fbf53723640 1 -- 172.21.15.120:0/2086785919 >> 172.21.15.120:0/2086785919 conn(0x7fbf4c06d1c0 msgr2=0x7fbf4c133dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:35.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.365+0000 7fbf53723640 1 -- 172.21.15.120:0/2086785919 shutdown_connections 2024-02-06T21:41:35.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.365+0000 7fbf53723640 1 -- 172.21.15.120:0/2086785919 wait complete. 2024-02-06T21:41:35.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.365+0000 7fbf53723640 1 librados: shutdown 2024-02-06T21:41:35.379 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-02-06T21:41:35.379 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-06T21:41:35.380 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-02-06T21:41:35.380 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-02-06T21:41:35.380 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-02-06T21:41:35.491 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.489+0000 7f123fa8f640 10 monclient: get_monmap_and_config 2024-02-06T21:41:35.491 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.489+0000 7f123fa8f640 10 monclient: build_initial_monmap 2024-02-06T21:41:35.491 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.489+0000 7f123fa8f640 10 monclient: monmap: 2024-02-06T21:41:35.491 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:35.491 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:35.491 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:35.491145+0000 2024-02-06T21:41:35.492 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:35.491145+0000 2024-02-06T21:41:35.492 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:35.492 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:35.492 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:35.492 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:35.492 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:35.492 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:35.492 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:35.492 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:35.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.490+0000 7f123fa8f640 1 Processor -- start 2024-02-06T21:41:35.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.490+0000 7f123fa8f640 1 -- start start 2024-02-06T21:41:35.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.490+0000 7f123fa8f640 10 monclient: init 2024-02-06T21:41:35.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.491+0000 7f123fa8f640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:35.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.491+0000 7f123fa8f640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-02-06T21:41:35.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.491+0000 7f123fa8f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12380a9c80 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:35.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.491+0000 7f123fa8f640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f12380aa060 0x7f123813f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:35.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.491+0000 7f123fa8f640 10 monclient(hunting): picked mon.noname-c con 0x7f12380aa060 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:35.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.491+0000 7f123fa8f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f123813fd50 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:35.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.491+0000 7f123fa8f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1238142720 con 0x7f123813fd50 2024-02-06T21:41:35.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.491+0000 7f123fa8f640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12380ab120 con 0x7f12380a9c80 2024-02-06T21:41:35.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.491+0000 7f123fa8f640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:35.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.491+0000 7f123fa8f640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12380ab390 con 0x7f12380aa060 2024-02-06T21:41:35.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.491+0000 7f123fa8f640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:35.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.491+0000 7f123fa8f640 10 monclient(hunting): authenticate will time out at 1325.828793s 2024-02-06T21:41:35.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.491+0000 7f123e02c640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f123813fd50 0x7f123814ce80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:52974/0 (socket says 172.21.15.120:52974) 2024-02-06T21:41:35.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.491+0000 7f123e02c640 1 -- 172.21.15.120:0/3921039229 learned_addr learned my addr 172.21.15.120:0/3921039229 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:41:35.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.492+0000 7f123d02a640 1 --2- 172.21.15.120:0/3921039229 >> v2:172.21.15.135:3300/0 conn(0x7f12380aa060 0x7f123813f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:35.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.492+0000 7f123d02a640 10 monclient(hunting): get_auth_request con 0x7f12380aa060 auth_method 0 2024-02-06T21:41:35.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.492+0000 7f123d02a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:35.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.492+0000 7f123d02a640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:35.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.492+0000 7f123d02a640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:35.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.493+0000 7f123d02a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:35.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.493+0000 7f123d02a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:35.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.493+0000 7f123d02a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:35.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.495+0000 7f123c829640 1 -- 172.21.15.120:0/3921039229 <== mon.0 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497085681 0 0) 0x7f1238142720 con 0x7f123813fd50 2024-02-06T21:41:35.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.495+0000 7f123c829640 10 monclient(hunting): _negotiate 2024-02-06T21:41:35.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.495+0000 7f123c829640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:35.497 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.495+0000 7f123c829640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:35.497 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.495+0000 7f123c829640 10 monclient(hunting): my global_id is 4269 2024-02-06T21:41:35.497 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.495+0000 7f123c829640 1 -- 172.21.15.120:0/3921039229 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1224003520 con 0x7f123813fd50 2024-02-06T21:41:35.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.496+0000 7f123d02a640 10 monclient(hunting): handle_auth_done global_id 4267 payload 274 2024-02-06T21:41:35.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.496+0000 7f123d02a640 1 -- 172.21.15.120:0/3921039229 >> v1:172.21.15.135:6789/0 conn(0x7f12380a9c80 legacy=0x7f1238142bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:35.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.496+0000 7f123d02a640 1 -- 172.21.15.120:0/3921039229 >> v1:172.21.15.120:6789/0 conn(0x7f123813fd50 legacy=0x7f123814ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:35.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.496+0000 7f123d02a640 10 monclient: _finish_hunting 0 2024-02-06T21:41:35.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.496+0000 7f123d02a640 1 monclient: found mon.noname-c 2024-02-06T21:41:35.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.496+0000 7f123d02a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:35.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.496+0000 7f123d02a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T21:41:35.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.496+0000 7f123d02a640 1 -- 172.21.15.120:0/3921039229 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12380ab670 con 0x7f12380aa060 2024-02-06T21:41:35.501 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.500+0000 7f123d02a640 1 --2- 172.21.15.120:0/3921039229 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12380aa060 0x7f123813f810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f12280097a0 tx=0x7f122802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f33ba3f6d4289608 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:35.506 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123c829640 1 -- 172.21.15.120:0/3921039229 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1228030610 con 0x7f12380aa060 2024-02-06T21:41:35.506 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123c829640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:35.507 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123c829640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:35.507 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123c829640 10 monclient: dump: 2024-02-06T21:41:35.507 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:35.507 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:35.507 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:35.507 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:35.507 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:35.507 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:35.507 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:35.507 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:35.507 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:35.507 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:35.507 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123c829640 10 monclient: _finish_auth 0 2024-02-06T21:41:35.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123c829640 10 monclient: _check_auth_tickets 2024-02-06T21:41:35.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123c829640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:35.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123fa8f640 5 monclient: authenticate success, global_id 4267 2024-02-06T21:41:35.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123c829640 1 -- 172.21.15.120:0/3921039229 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1228030770 con 0x7f12380aa060 2024-02-06T21:41:35.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123fa8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:35.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123c829640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:35.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123fa8f640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:35.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123fa8f640 4 set_mon_vals no callback set 2024-02-06T21:41:35.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123c829640 1 -- 172.21.15.120:0/3921039229 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1228030b00 con 0x7f12380aa060 2024-02-06T21:41:35.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123c829640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:35.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123c829640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:35.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123c829640 10 monclient: dump: 2024-02-06T21:41:35.508 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:35.509 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:35.509 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:35.509 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:35.509 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:35.509 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:35.509 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:35.509 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:35.509 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:35.509 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:35.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123fa8f640 10 monclient: shutdown 2024-02-06T21:41:35.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123fa8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:35.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123fa8f640 1 -- 172.21.15.120:0/3921039229 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12380aa060 msgr2=0x7f123813f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:35.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123fa8f640 1 --2- 172.21.15.120:0/3921039229 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12380aa060 0x7f123813f810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f12280097a0 tx=0x7f122802b960 comp rx=0 tx=0).stop 2024-02-06T21:41:35.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123fa8f640 1 -- 172.21.15.120:0/3921039229 shutdown_connections 2024-02-06T21:41:35.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123fa8f640 1 --2- 172.21.15.120:0/3921039229 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12380aa060 0x7f123813f810 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:35.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.505+0000 7f123fa8f640 1 -- 172.21.15.120:0/3921039229 >> 172.21.15.120:0/3921039229 conn(0x7f12380b4ba0 msgr2=0x7f12380b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:35.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.506+0000 7f123fa8f640 1 -- 172.21.15.120:0/3921039229 shutdown_connections 2024-02-06T21:41:35.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.506+0000 7f123fa8f640 1 -- 172.21.15.120:0/3921039229 wait complete. 2024-02-06T21:41:35.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.506+0000 7f123fa8f640 10 monclient: build_initial_monmap 2024-02-06T21:41:35.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.506+0000 7f123fa8f640 10 monclient: monmap: 2024-02-06T21:41:35.513 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:35.513 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:35.513 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:35.513 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:35.514 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:35.514 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:35.514 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:35.514 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:35.514 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:35.514 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:35.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.506+0000 7f123fa8f640 1 librados: starting msgr at 2024-02-06T21:41:35.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.506+0000 7f123fa8f640 1 librados: starting objecter 2024-02-06T21:41:35.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 1 Processor -- start 2024-02-06T21:41:35.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 1 -- start start 2024-02-06T21:41:35.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 1 librados: setting wanted keys 2024-02-06T21:41:35.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 1 librados: calling monclient init 2024-02-06T21:41:35.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 10 monclient: init 2024-02-06T21:41:35.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:35.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T21:41:35.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12380a9c80 0x7f1238144510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:35.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 10 monclient(hunting): picked mon.noname-b con 0x7f12380a9c80 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:35.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12380aa060 0x7f1238144fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:35.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 10 monclient(hunting): picked mon.noname-a con 0x7f12380aa060 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:35.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f123813fd50 0x7f1238136330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:35.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 10 monclient(hunting): picked mon.noname-c con 0x7f123813fd50 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:35.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:35.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1238140790 con 0x7f12380aa060 2024-02-06T21:41:35.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:35.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1238146340 con 0x7f12380a9c80 2024-02-06T21:41:35.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:35.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1238146620 con 0x7f123813fd50 2024-02-06T21:41:35.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:35.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.507+0000 7f123fa8f640 10 monclient(hunting): authenticate will time out at 1325.845135s 2024-02-06T21:41:35.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.508+0000 7f123d02a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12380aa060 0x7f1238144fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:35.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.508+0000 7f123e02c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f123813fd50 0x7f1238136330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:35.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.508+0000 7f123e02c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f123813fd50 0x7f1238136330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:56734/0 (socket says 172.21.15.120:56734) 2024-02-06T21:41:35.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.508+0000 7f123e02c640 1 -- 172.21.15.120:0/3080087997 learned_addr learned my addr 172.21.15.120:0/3080087997 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:35.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.508+0000 7f123e02c640 10 monclient(hunting): get_auth_request con 0x7f123813fd50 auth_method 0 2024-02-06T21:41:35.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.508+0000 7f123e02c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:35.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.508+0000 7f123e02c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:35.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.508+0000 7f123e02c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:35.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.508+0000 7f123d02a640 10 monclient(hunting): get_auth_request con 0x7f12380aa060 auth_method 0 2024-02-06T21:41:35.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.508+0000 7f123d02a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:35.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.508+0000 7f123d02a640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:35.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.508+0000 7f123d02a640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:35.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.508+0000 7f123d82b640 1 --2- 172.21.15.120:0/3080087997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12380a9c80 0x7f1238144510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:35.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.509+0000 7f123e02c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:35.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.509+0000 7f123e02c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:35.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.509+0000 7f123e02c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:35.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.509+0000 7f123d82b640 10 monclient(hunting): get_auth_request con 0x7f12380a9c80 auth_method 0 2024-02-06T21:41:35.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.509+0000 7f123d82b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:35.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.509+0000 7f123d82b640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:35.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.509+0000 7f123d82b640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:35.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.509+0000 7f123d02a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:35.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.509+0000 7f123d02a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:35.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.509+0000 7f123d02a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:35.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.509+0000 7f123d82b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:35.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.509+0000 7f123d82b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:35.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.510+0000 7f123d82b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:35.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.511+0000 7f123e02c640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2024-02-06T21:41:35.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.511+0000 7f123e02c640 1 -- 172.21.15.120:0/3080087997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12380a9c80 msgr2=0x7f1238144510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:35.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.511+0000 7f123e02c640 1 --2- 172.21.15.120:0/3080087997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12380a9c80 0x7f1238144510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:35.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.511+0000 7f123e02c640 1 -- 172.21.15.120:0/3080087997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12380aa060 msgr2=0x7f1238144fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:35.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.511+0000 7f123e02c640 1 --2- 172.21.15.120:0/3080087997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12380aa060 0x7f1238144fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:35.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.511+0000 7f123e02c640 10 monclient: _finish_hunting 0 2024-02-06T21:41:35.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.511+0000 7f123e02c640 1 monclient: found mon.noname-c 2024-02-06T21:41:35.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.511+0000 7f123e02c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:35.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.511+0000 7f123e02c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:41:35.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.511+0000 7f123e02c640 1 -- 172.21.15.120:0/3080087997 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1228009400 con 0x7f123813fd50 2024-02-06T21:41:35.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.514+0000 7f123e02c640 1 --2- 172.21.15.120:0/3080087997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f123813fd50 0x7f1238136330 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f123000caa0 tx=0x7f123000cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:35.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.514+0000 7f122e7fc640 10 client.?.objecter ms_handle_connect 0x7f123813fd50 2024-02-06T21:41:35.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.514+0000 7f122e7fc640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:35.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f122e7fc640 1 -- 172.21.15.120:0/3080087997 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123000ac00 con 0x7f123813fd50 2024-02-06T21:41:35.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f122e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:35.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f122e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:35.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f122e7fc640 10 monclient: dump: 2024-02-06T21:41:35.522 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:35.522 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:35.522 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:35.522 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:35.522 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:35.522 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:35.522 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:35.522 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:35.522 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:35.522 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:35.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f122e7fc640 10 monclient: _finish_auth 0 2024-02-06T21:41:35.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f122e7fc640 10 monclient: _check_auth_tickets 2024-02-06T21:41:35.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f122e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:35.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f122e7fc640 1 -- 172.21.15.120:0/3080087997 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123000ad60 con 0x7f123813fd50 2024-02-06T21:41:35.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f122e7fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:35.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f122e7fc640 1 -- 172.21.15.120:0/3080087997 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230012620 con 0x7f123813fd50 2024-02-06T21:41:35.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f123fa8f640 5 monclient: authenticate success, global_id 4247 2024-02-06T21:41:35.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f122e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:35.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f122e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:35.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f122e7fc640 10 monclient: dump: 2024-02-06T21:41:35.523 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:35.523 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:35.523 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:35.523 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:35.523 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:35.523 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:35.524 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:35.524 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:35.524 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:35.524 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:35.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f122ffff640 4 set_mon_vals no callback set 2024-02-06T21:41:35.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f123fa8f640 10 monclient: _renew_subs 2024-02-06T21:41:35.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f123fa8f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:35.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f123fa8f640 1 -- 172.21.15.120:0/3080087997 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12381454f0 con 0x7f123813fd50 2024-02-06T21:41:35.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f123fa8f640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:35.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f123fa8f640 10 monclient: _renew_subs 2024-02-06T21:41:35.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f123fa8f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:35.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f123fa8f640 1 -- 172.21.15.120:0/3080087997 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1238136a50 con 0x7f123813fd50 2024-02-06T21:41:35.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.518+0000 7f123fa8f640 1 librados: init done 2024-02-06T21:41:35.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.519+0000 7f123fa8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:35.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.519+0000 7f123fa8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:35.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.519+0000 7f123fa8f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:35.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.519+0000 7f123fa8f640 1 -- 172.21.15.120:0/3080087997 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f123804ef90 con 0x7f123813fd50 2024-02-06T21:41:35.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.799+0000 7f122e7fc640 1 -- 172.21.15.120:0/3080087997 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 4) v1 ==== 89732+0+0 (secure 0 0 0) 0x7f1230007070 con 0x7f123813fd50 2024-02-06T21:41:35.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.799+0000 7f122e7fc640 1 --2- 172.21.15.120:0/3080087997 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f121c06d160 0x7f121c06f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:35.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.800+0000 7f123d82b640 1 --2- 172.21.15.120:0/3080087997 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f121c06d160 0x7f121c06f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:35.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.800+0000 7f123d82b640 10 monclient: get_auth_request con 0x7f121c06d160 auth_method 0 2024-02-06T21:41:35.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.800+0000 7f122e7fc640 10 client.4247.objecter ms_handle_connect 0x7f121c06d160 2024-02-06T21:41:35.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.800+0000 7f123d82b640 1 --2- 172.21.15.120:0/3080087997 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f121c06d160 0x7f121c06f620 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1234001780 tx=0x7f1234009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:35.820 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.818+0000 7f122e7fc640 1 -- 172.21.15.120:0/3080087997 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1230058d70 con 0x7f123813fd50 2024-02-06T21:41:35.820 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.818+0000 7f122e7fc640 10 client.4247.objecter ms_dispatch 0x7f1238000b90 osd_map(22..22 src has 1..22) v4 2024-02-06T21:41:35.820 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.818+0000 7f122e7fc640 3 client.4247.objecter handle_osd_map got epochs [22,22] > 0 2024-02-06T21:41:35.820 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.818+0000 7f122e7fc640 3 client.4247.objecter handle_osd_map decoding full epoch 22 2024-02-06T21:41:35.820 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.818+0000 7f122e7fc640 20 client.4247.objecter dump_active .. 0 homeless 2024-02-06T21:41:35.820 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.818+0000 7f122e7fc640 1 -- 172.21.15.120:0/3080087997 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1230055010 con 0x7f123813fd50 2024-02-06T21:41:35.820 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.818+0000 7f122e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:35.820 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:35.818+0000 7f122e7fc640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:36.078 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.077+0000 7f123fa8f640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-02-06T21:41:36.079 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.077+0000 7f123fa8f640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-02-06T21:41:36.079 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.077+0000 7f123fa8f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:36.079 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.077+0000 7f123fa8f640 1 -- 172.21.15.120:0/3080087997 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f1238146760 con 0x7f123813fd50 2024-02-06T21:41:36.260 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.259+0000 7f122e7fc640 1 -- 172.21.15.120:0/3080087997 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+171704 (secure 0 0 0) 0x7f1238146760 con 0x7f123813fd50 2024-02-06T21:41:36.260 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.259+0000 7f122e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-02-06T21:41:36.260 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.259+0000 7f122e7fc640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:36.263 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:41:36.265 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.261+0000 7f123fa8f640 10 librados: watch_flush enter 2024-02-06T21:41:36.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.261+0000 7f123fa8f640 10 librados: watch_flush exit 2024-02-06T21:41:36.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.261+0000 7f123fa8f640 20 client.4247.objecter shutdown clearing up homeless session... 2024-02-06T21:41:36.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.261+0000 7f123fa8f640 10 client.4247.objecter successfully canceled tick 2024-02-06T21:41:36.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.261+0000 7f123fa8f640 1 -- 172.21.15.120:0/3080087997 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f121c06d160 msgr2=0x7f121c06f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:36.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.261+0000 7f123fa8f640 1 --2- 172.21.15.120:0/3080087997 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f121c06d160 0x7f121c06f620 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1234001780 tx=0x7f1234009040 comp rx=0 tx=0).stop 2024-02-06T21:41:36.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.261+0000 7f123fa8f640 10 monclient: shutdown 2024-02-06T21:41:36.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.261+0000 7f123fa8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:36.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.261+0000 7f123fa8f640 1 -- 172.21.15.120:0/3080087997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f123813fd50 msgr2=0x7f1238136330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:36.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.261+0000 7f123fa8f640 1 --2- 172.21.15.120:0/3080087997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f123813fd50 0x7f1238136330 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f123000caa0 tx=0x7f123000cf60 comp rx=0 tx=0).stop 2024-02-06T21:41:36.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.262+0000 7f123fa8f640 1 -- 172.21.15.120:0/3080087997 shutdown_connections 2024-02-06T21:41:36.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.262+0000 7f123fa8f640 1 --2- 172.21.15.120:0/3080087997 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f121c06d160 0x7f121c06f620 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:36.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.262+0000 7f123fa8f640 1 --2- 172.21.15.120:0/3080087997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f123813fd50 0x7f1238136330 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:36.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.262+0000 7f123fa8f640 1 --2- 172.21.15.120:0/3080087997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12380aa060 0x7f1238144fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:36.267 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.262+0000 7f123fa8f640 1 --2- 172.21.15.120:0/3080087997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12380a9c80 0x7f1238144510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:36.267 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.262+0000 7f123fa8f640 1 -- 172.21.15.120:0/3080087997 >> 172.21.15.120:0/3080087997 conn(0x7f12380b4ba0 msgr2=0x7f12380b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:36.267 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.262+0000 7f123fa8f640 1 -- 172.21.15.120:0/3080087997 shutdown_connections 2024-02-06T21:41:36.267 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.262+0000 7f123fa8f640 1 -- 172.21.15.120:0/3080087997 wait complete. 2024-02-06T21:41:36.267 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.262+0000 7f123fa8f640 1 librados: shutdown 2024-02-06T21:41:36.274 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":315664144},{"type":"v1","addr":"172.21.15.135:6801","nonce":315664144}]},"active_addr":"172.21.15.135:6801/315664144","active_change":"2024-02-06T21:40:37.319386+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4108,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.135:0","nonce":124048395}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.135:0","nonce":2569651401}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.135:0","nonce":2690624901}]}]} 2024-02-06T21:41:36.281 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-02-06T21:41:36.281 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-02-06T21:41:36.281 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-06T21:41:36.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.392+0000 7fd8362f6640 10 monclient: get_monmap_and_config 2024-02-06T21:41:36.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.392+0000 7fd8362f6640 10 monclient: build_initial_monmap 2024-02-06T21:41:36.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.392+0000 7fd8362f6640 10 monclient: monmap: 2024-02-06T21:41:36.395 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:36.395 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:36.395 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:36.394174+0000 2024-02-06T21:41:36.395 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:36.394174+0000 2024-02-06T21:41:36.395 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:36.395 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:36.395 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:36.395 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:36.395 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:36.395 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:36.396 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:36.396 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:36.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.393+0000 7fd8362f6640 1 Processor -- start 2024-02-06T21:41:36.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.393+0000 7fd8362f6640 1 -- start start 2024-02-06T21:41:36.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.393+0000 7fd8362f6640 10 monclient: init 2024-02-06T21:41:36.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.394+0000 7fd8362f6640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:36.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.394+0000 7fd8362f6640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-02-06T21:41:36.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.394+0000 7fd8362f6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8300b54b0 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:36.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.394+0000 7fd8362f6640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd83013ff40 0x7fd8300a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:36.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.394+0000 7fd8362f6640 10 monclient(hunting): picked mon.noname-a con 0x7fd83013ff40 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:36.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.394+0000 7fd8362f6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd830142680 0x7fd830142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:36.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.394+0000 7fd8362f6640 10 monclient(hunting): picked mon.noname-b con 0x7fd830142680 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:36.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.394+0000 7fd8362f6640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8300ab720 con 0x7fd8300b54b0 2024-02-06T21:41:36.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.394+0000 7fd8362f6640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:36.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.394+0000 7fd8362f6640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8300ab520 con 0x7fd83013ff40 2024-02-06T21:41:36.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.394+0000 7fd8362f6640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:36.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.394+0000 7fd8362f6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8300b5890 con 0x7fd830142680 2024-02-06T21:41:36.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.394+0000 7fd8362f6640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:36.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.394+0000 7fd8362f6640 10 monclient(hunting): authenticate will time out at 1326.731887s 2024-02-06T21:41:36.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.395+0000 7fd82ffff640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fd8300b54b0 0x7fd83013f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.120:58764/0 (socket says 172.21.15.120:58764) 2024-02-06T21:41:36.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.395+0000 7fd82ffff640 1 -- 172.21.15.120:0/2714450361 learned_addr learned my addr 172.21.15.120:0/2714450361 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:41:36.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.410+0000 7fd82effd640 1 -- 172.21.15.120:0/2714450361 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877109296 0 0) 0x7fd8300ab720 con 0x7fd8300b54b0 2024-02-06T21:41:36.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.410+0000 7fd82effd640 10 monclient(hunting): _negotiate 2024-02-06T21:41:36.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.410+0000 7fd82effd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:36.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.410+0000 7fd82effd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:36.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.410+0000 7fd82effd640 10 monclient(hunting): my global_id is 4273 2024-02-06T21:41:36.413 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.410+0000 7fd82effd640 1 -- 172.21.15.120:0/2714450361 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd818003520 con 0x7fd8300b54b0 2024-02-06T21:41:36.416 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.415+0000 7fd82f7fe640 1 --2- 172.21.15.120:0/2714450361 >> v2:172.21.15.120:3300/0 conn(0x7fd83013ff40 0x7fd8300a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:36.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.416+0000 7fd82f7fe640 10 monclient(hunting): get_auth_request con 0x7fd83013ff40 auth_method 0 2024-02-06T21:41:36.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.416+0000 7fd82f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:36.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.416+0000 7fd82f7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:36.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.416+0000 7fd82f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:36.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.416+0000 7fd82f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:36.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.416+0000 7fd82f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:36.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.416+0000 7fd82f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:36.419 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.418+0000 7fd82f7fe640 10 monclient(hunting): handle_auth_done global_id 4275 payload 274 2024-02-06T21:41:36.419 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.418+0000 7fd82f7fe640 1 -- 172.21.15.120:0/2714450361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd830142680 msgr2=0x7fd830142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:36.419 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.418+0000 7fd82f7fe640 1 --2- 172.21.15.120:0/2714450361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd830142680 0x7fd830142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:36.419 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.418+0000 7fd82f7fe640 1 -- 172.21.15.120:0/2714450361 >> v1:172.21.15.135:6789/0 conn(0x7fd8300b54b0 legacy=0x7fd83013f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:36.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.418+0000 7fd82f7fe640 10 monclient: _finish_hunting 0 2024-02-06T21:41:36.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.418+0000 7fd82f7fe640 1 monclient: found mon.noname-a 2024-02-06T21:41:36.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.418+0000 7fd82f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:36.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.418+0000 7fd82f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:36.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.418+0000 7fd82f7fe640 1 -- 172.21.15.120:0/2714450361 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8300b4800 con 0x7fd83013ff40 2024-02-06T21:41:36.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.420+0000 7fd82f7fe640 1 --2- 172.21.15.120:0/2714450361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd83013ff40 0x7fd8300a9860 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd8240097d0 tx=0x7fd82402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea77642edeed63d server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:36.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd82effd640 1 -- 172.21.15.120:0/2714450361 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824031800 con 0x7fd83013ff40 2024-02-06T21:41:36.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd82effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:36.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd82effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:36.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd82effd640 10 monclient: dump: 2024-02-06T21:41:36.428 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:36.428 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:36.428 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:36.428 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:36.428 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:36.428 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:36.428 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:36.429 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:36.429 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:36.429 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:36.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd82effd640 10 monclient: _finish_auth 0 2024-02-06T21:41:36.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd82effd640 10 monclient: _check_auth_tickets 2024-02-06T21:41:36.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd82effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:36.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd82effd640 1 -- 172.21.15.120:0/2714450361 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd824031960 con 0x7fd83013ff40 2024-02-06T21:41:36.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd8362f6640 5 monclient: authenticate success, global_id 4275 2024-02-06T21:41:36.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd8362f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:36.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd82effd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:36.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd8362f6640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:36.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd8362f6640 4 set_mon_vals no callback set 2024-02-06T21:41:36.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd82effd640 1 -- 172.21.15.120:0/2714450361 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824031cf0 con 0x7fd83013ff40 2024-02-06T21:41:36.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd82effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:36.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd82effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:36.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.426+0000 7fd82effd640 10 monclient: dump: 2024-02-06T21:41:36.430 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:36.430 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:36.430 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:36.430 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:36.430 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:36.430 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:36.430 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:36.430 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:36.430 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:36.430 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:36.433 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.427+0000 7fd8362f6640 10 monclient: shutdown 2024-02-06T21:41:36.433 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.427+0000 7fd8362f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:36.433 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.427+0000 7fd8362f6640 1 -- 172.21.15.120:0/2714450361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd83013ff40 msgr2=0x7fd8300a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:36.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.427+0000 7fd8362f6640 1 --2- 172.21.15.120:0/2714450361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd83013ff40 0x7fd8300a9860 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd8240097d0 tx=0x7fd82402b8a0 comp rx=0 tx=0).stop 2024-02-06T21:41:36.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.427+0000 7fd8362f6640 1 -- 172.21.15.120:0/2714450361 shutdown_connections 2024-02-06T21:41:36.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.427+0000 7fd8362f6640 1 --2- 172.21.15.120:0/2714450361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd830142680 0x7fd830142ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:36.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.427+0000 7fd8362f6640 1 --2- 172.21.15.120:0/2714450361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd83013ff40 0x7fd8300a9860 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:36.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.427+0000 7fd8362f6640 1 -- 172.21.15.120:0/2714450361 >> 172.21.15.120:0/2714450361 conn(0x7fd8300b8660 msgr2=0x7fd8300b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:36.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.427+0000 7fd8362f6640 1 -- 172.21.15.120:0/2714450361 shutdown_connections 2024-02-06T21:41:36.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.427+0000 7fd8362f6640 1 -- 172.21.15.120:0/2714450361 wait complete. 2024-02-06T21:41:36.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.428+0000 7fd8362f6640 10 monclient: build_initial_monmap 2024-02-06T21:41:36.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.428+0000 7fd8362f6640 10 monclient: monmap: 2024-02-06T21:41:36.434 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:36.434 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:36.434 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:36.434 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:36.434 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:36.434 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:36.435 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:36.435 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:36.435 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:36.435 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:36.435 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.428+0000 7fd8362f6640 1 librados: starting msgr at 2024-02-06T21:41:36.435 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.428+0000 7fd8362f6640 1 librados: starting objecter 2024-02-06T21:41:36.435 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.428+0000 7fd8362f6640 1 Processor -- start 2024-02-06T21:41:36.435 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 1 -- start start 2024-02-06T21:41:36.435 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 1 librados: setting wanted keys 2024-02-06T21:41:36.435 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 1 librados: calling monclient init 2024-02-06T21:41:36.435 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 10 monclient: init 2024-02-06T21:41:36.435 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:36.435 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T21:41:36.435 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8300b54b0 0x7fd830137900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:36.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 10 monclient(hunting): picked mon.noname-c con 0x7fd8300b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:36.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd83013ff40 0x7fd8300b7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:36.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 10 monclient(hunting): picked mon.noname-b con 0x7fd83013ff40 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:36.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd830142680 0x7fd830138450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:36.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 10 monclient(hunting): picked mon.noname-a con 0x7fd830142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:36.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:36.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd830133dd0 con 0x7fd830142680 2024-02-06T21:41:36.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:36.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd830137ee0 con 0x7fd83013ff40 2024-02-06T21:41:36.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:36.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8300ac4c0 con 0x7fd8300b54b0 2024-02-06T21:41:36.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:36.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.429+0000 7fd8362f6640 10 monclient(hunting): authenticate will time out at 1326.767507s 2024-02-06T21:41:36.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.430+0000 7fd834893640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd830142680 0x7fd830138450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:36.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.430+0000 7fd82f7fe640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd83013ff40 0x7fd8300b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:36.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd834893640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd830142680 0x7fd830138450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60066/0 (socket says 172.21.15.120:60066) 2024-02-06T21:41:36.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd834893640 1 -- 172.21.15.120:0/2875509150 learned_addr learned my addr 172.21.15.120:0/2875509150 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:36.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd834893640 10 monclient(hunting): get_auth_request con 0x7fd830142680 auth_method 0 2024-02-06T21:41:36.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd834893640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:36.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd834893640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:36.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd834893640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:36.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd82f7fe640 10 monclient(hunting): get_auth_request con 0x7fd83013ff40 auth_method 0 2024-02-06T21:41:36.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd82f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:36.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd82f7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:36.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd82f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:36.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd82f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:36.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd82f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:36.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd82f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:36.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd834893640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:36.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd834893640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:36.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.431+0000 7fd834893640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:36.439 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.434+0000 7fd82f7fe640 10 monclient(hunting): handle_auth_done global_id 4276 payload 824 2024-02-06T21:41:36.439 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.434+0000 7fd82f7fe640 1 -- 172.21.15.120:0/2875509150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8300b54b0 msgr2=0x7fd830137900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:36.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.434+0000 7fd82f7fe640 1 --2- 172.21.15.120:0/2875509150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8300b54b0 0x7fd830137900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:36.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.434+0000 7fd82f7fe640 1 -- 172.21.15.120:0/2875509150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd830142680 msgr2=0x7fd830138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:36.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.434+0000 7fd82f7fe640 1 --2- 172.21.15.120:0/2875509150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd830142680 0x7fd830138450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:36.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.434+0000 7fd82f7fe640 10 monclient: _finish_hunting 0 2024-02-06T21:41:36.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.434+0000 7fd82f7fe640 1 monclient: found mon.noname-b 2024-02-06T21:41:36.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.434+0000 7fd82f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:36.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.434+0000 7fd82f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:36.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.434+0000 7fd82f7fe640 1 -- 172.21.15.120:0/2875509150 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd824009400 con 0x7fd83013ff40 2024-02-06T21:41:36.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.437+0000 7fd82f7fe640 1 --2- 172.21.15.120:0/2875509150 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd83013ff40 0x7fd8300b7140 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd824033000 tx=0x7fd8240046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:36.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.437+0000 7fd82cff9640 10 client.?.objecter ms_handle_connect 0x7fd83013ff40 2024-02-06T21:41:36.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.437+0000 7fd82cff9640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:36.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd82cff9640 1 -- 172.21.15.120:0/2875509150 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82403bc70 con 0x7fd83013ff40 2024-02-06T21:41:36.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd82cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:36.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd82cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:36.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd82cff9640 10 monclient: dump: 2024-02-06T21:41:36.443 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:36.443 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:36.443 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:36.443 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:36.443 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:36.443 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:36.444 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:36.444 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:36.444 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:36.444 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:36.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd82cff9640 10 monclient: _finish_auth 0 2024-02-06T21:41:36.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd82cff9640 10 monclient: _check_auth_tickets 2024-02-06T21:41:36.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd82cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:36.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd82cff9640 1 -- 172.21.15.120:0/2875509150 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82403be10 con 0x7fd83013ff40 2024-02-06T21:41:36.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd8362f6640 5 monclient: authenticate success, global_id 4276 2024-02-06T21:41:36.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd82cff9640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:36.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd8362f6640 10 monclient: _renew_subs 2024-02-06T21:41:36.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd82cff9640 1 -- 172.21.15.120:0/2875509150 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824045630 con 0x7fd83013ff40 2024-02-06T21:41:36.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd8362f6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:36.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd8362f6640 1 -- 172.21.15.120:0/2875509150 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd830139340 con 0x7fd83013ff40 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd82cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd82cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd82cff9640 10 monclient: dump: 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd82e7fc640 4 set_mon_vals no callback set 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd8362f6640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:36.445 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd8362f6640 10 monclient: _renew_subs 2024-02-06T21:41:36.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd8362f6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:36.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd8362f6640 1 -- 172.21.15.120:0/2875509150 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8300acac0 con 0x7fd83013ff40 2024-02-06T21:41:36.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.441+0000 7fd8362f6640 1 librados: init done 2024-02-06T21:41:36.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.442+0000 7fd8362f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:36.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.442+0000 7fd8362f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:36.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.442+0000 7fd8362f6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:36.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.442+0000 7fd8362f6640 1 -- 172.21.15.120:0/2875509150 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd830138170 con 0x7fd83013ff40 2024-02-06T21:41:36.462 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.460+0000 7fd82cff9640 1 -- 172.21.15.120:0/2875509150 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 4) v1 ==== 89732+0+0 (secure 0 0 0) 0x7fd82404e450 con 0x7fd83013ff40 2024-02-06T21:41:36.462 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.461+0000 7fd82cff9640 1 --2- 172.21.15.120:0/2875509150 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd81c06d140 0x7fd81c06f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:36.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.462+0000 7fd82cff9640 1 -- 172.21.15.120:0/2875509150 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd82402e070 con 0x7fd83013ff40 2024-02-06T21:41:36.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.462+0000 7fd82cff9640 10 client.4276.objecter ms_dispatch 0x7fd830000b90 osd_map(22..22 src has 1..22) v4 2024-02-06T21:41:36.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.462+0000 7fd82cff9640 3 client.4276.objecter handle_osd_map got epochs [22,22] > 0 2024-02-06T21:41:36.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.462+0000 7fd82cff9640 3 client.4276.objecter handle_osd_map decoding full epoch 22 2024-02-06T21:41:36.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.462+0000 7fd82cff9640 20 client.4276.objecter dump_active .. 0 homeless 2024-02-06T21:41:36.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.468+0000 7fd82ffff640 1 --2- 172.21.15.120:0/2875509150 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd81c06d140 0x7fd81c06f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:36.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.468+0000 7fd82ffff640 10 monclient: get_auth_request con 0x7fd81c06d140 auth_method 0 2024-02-06T21:41:36.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.468+0000 7fd82ffff640 1 --2- 172.21.15.120:0/2875509150 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd81c06d140 0x7fd81c06f600 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd820001610 tx=0x7fd820007930 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:36.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.468+0000 7fd82cff9640 10 client.4276.objecter ms_handle_connect 0x7fd81c06d140 2024-02-06T21:41:36.724 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.722+0000 7fd82cff9640 1 -- 172.21.15.120:0/2875509150 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd824077260 con 0x7fd83013ff40 2024-02-06T21:41:36.724 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.722+0000 7fd82cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:36.724 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.722+0000 7fd82cff9640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:36.900 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.899+0000 7fd8362f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:36.901 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.899+0000 7fd8362f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:36.901 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.899+0000 7fd8362f6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:36.901 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:36.899+0000 7fd8362f6640 1 -- 172.21.15.120:0/2875509150 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8300ac600 con 0x7fd83013ff40 2024-02-06T21:41:37.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.009+0000 7fd82cff9640 1 -- 172.21.15.120:0/2875509150 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd824077630 con 0x7fd83013ff40 2024-02-06T21:41:37.058 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.057+0000 7fd82cff9640 1 -- 172.21.15.120:0/2875509150 <== mon.1 v2:172.21.15.135:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14777 (secure 0 0 0) 0x7fd8240a8e30 con 0x7fd83013ff40 2024-02-06T21:41:37.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.057+0000 7fd82cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:37.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.057+0000 7fd82cff9640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:37.059 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:41:37.059 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":23,"fsid":"4e4791bf-0a02-4478-9517-2d2e893f4f7c","created":"2024-02-06T21:40:31.800075+0000","modified":"2024-02-06T21:41:36.314825+0000","last_up_change":"2024-02-06T21:41:29.190309+0000","last_in_change":"2024-02-06T21:40:37.354288+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-06T21:41:20.093970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-02-06T21:41:33.186378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4bd0f591-2cf1-42da-8ac1-dce3188580c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6817","nonce":2062960776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6819","nonce":2062960776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6823","nonce":2062960776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6821","nonce":2062960776}]},"public_addr":"172.21.15.120:6817/2062960776","cluster_addr":"172.21.15.120:6819/2062960776","heartbeat_back_addr":"172.21.15.120:6823/2062960776","heartbeat_front_addr":"172.21.15.120:6821/2062960776","state":["exists","up"]},{"osd":1,"uuid":"81dc801c-3768-4c2c-9adb-6c8300a341f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6809","nonce":3348315891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6811","nonce":3348315891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6815","nonce":3348315891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6813","nonce":3348315891}]},"public_addr":"172.21.15.120:6809/3348315891","cluster_addr":"172.21.15.120:6811/3348315891","heartbeat_back_addr":"172.21.15.120:6815/3348315891","heartbeat_front_addr":"172.21.15.120:6813/3348315891","state":["exists","up"]},{"osd":2,"uuid":"9289721b-c13f-4a17-be68-3698090f69af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6825","nonce":3229495265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6827","nonce":3229495265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6831","nonce":3229495265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6829","nonce":3229495265}]},"public_addr":"172.21.15.120:6825/3229495265","cluster_addr":"172.21.15.120:6827/3229495265","heartbeat_back_addr":"172.21.15.120:6831/3229495265","heartbeat_front_addr":"172.21.15.120:6829/3229495265","state":["exists","up"]},{"osd":3,"uuid":"53534db0-2d4d-44ed-90e8-d67ea878b51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6801","nonce":3324566974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6803","nonce":3324566974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6807","nonce":3324566974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6805","nonce":3324566974}]},"public_addr":"172.21.15.120:6801/3324566974","cluster_addr":"172.21.15.120:6803/3324566974","heartbeat_back_addr":"172.21.15.120:6807/3324566974","heartbeat_front_addr":"172.21.15.120:6805/3324566974","state":["exists","up"]},{"osd":4,"uuid":"e4205ab5-787c-44e0-9b96-273b8c9cf1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6803","nonce":2191547918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6805","nonce":2191547918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6809","nonce":2191547918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6807","nonce":2191547918}]},"public_addr":"172.21.15.135:6803/2191547918","cluster_addr":"172.21.15.135:6805/2191547918","heartbeat_back_addr":"172.21.15.135:6809/2191547918","heartbeat_front_addr":"172.21.15.135:6807/2191547918","state":["exists","up"]},{"osd":5,"uuid":"43b600bc-3afe-44bd-bd3f-690106e7e0ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6819","nonce":563193066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6821","nonce":563193066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6826","nonce":563193066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6823","nonce":563193066}]},"public_addr":"172.21.15.135:6819/563193066","cluster_addr":"172.21.15.135:6821/563193066","heartbeat_back_addr":"172.21.15.135:6826/563193066","heartbeat_front_addr":"172.21.15.135:6823/563193066","state":["exists","up"]},{"osd":6,"uuid":"1d6b9f96-1f9d-44f2-83b0-93f5c342fafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6811","nonce":1080459988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6813","nonce":1080459988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6817","nonce":1080459988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6815","nonce":1080459988}]},"public_addr":"172.21.15.135:6811/1080459988","cluster_addr":"172.21.15.135:6813/1080459988","heartbeat_back_addr":"172.21.15.135:6817/1080459988","heartbeat_front_addr":"172.21.15.135:6815/1080459988","state":["exists","up"]},{"osd":7,"uuid":"ecae00f5-8e29-47dd-90d5-e51503bc9dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6825","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6827","nonce":3740758337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6829","nonce":3740758337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6833","nonce":3740758337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6831","nonce":3740758337}]},"public_addr":"172.21.15.135:6827/3740758337","cluster_addr":"172.21.15.135:6829/3740758337","heartbeat_back_addr":"172.21.15.135:6833/3740758337","heartbeat_front_addr":"172.21.15.135:6831/3740758337","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.344503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:17.073640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.245112+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.202999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:15.825107+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:15.918704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.203923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:15.753628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-06T21:41:37.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 10 librados: watch_flush enter 2024-02-06T21:41:37.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 10 librados: watch_flush exit 2024-02-06T21:41:37.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 20 client.4276.objecter shutdown clearing up homeless session... 2024-02-06T21:41:37.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 10 client.4276.objecter successfully canceled tick 2024-02-06T21:41:37.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 1 -- 172.21.15.120:0/2875509150 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd81c06d140 msgr2=0x7fd81c06f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:37.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 1 --2- 172.21.15.120:0/2875509150 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd81c06d140 0x7fd81c06f600 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd820001610 tx=0x7fd820007930 comp rx=0 tx=0).stop 2024-02-06T21:41:37.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 10 monclient: shutdown 2024-02-06T21:41:37.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:37.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 1 -- 172.21.15.120:0/2875509150 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd83013ff40 msgr2=0x7fd8300b7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:37.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 1 --2- 172.21.15.120:0/2875509150 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd83013ff40 0x7fd8300b7140 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd824033000 tx=0x7fd8240046c0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 1 -- 172.21.15.120:0/2875509150 shutdown_connections 2024-02-06T21:41:37.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 1 --2- 172.21.15.120:0/2875509150 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd81c06d140 0x7fd81c06f600 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 1 --2- 172.21.15.120:0/2875509150 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd83013ff40 0x7fd8300b7140 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 1 --2- 172.21.15.120:0/2875509150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd830142680 0x7fd830138450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 1 --2- 172.21.15.120:0/2875509150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8300b54b0 0x7fd830137900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 1 -- 172.21.15.120:0/2875509150 >> 172.21.15.120:0/2875509150 conn(0x7fd8300b8660 msgr2=0x7fd83013b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:37.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 1 -- 172.21.15.120:0/2875509150 shutdown_connections 2024-02-06T21:41:37.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 1 -- 172.21.15.120:0/2875509150 wait complete. 2024-02-06T21:41:37.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.059+0000 7fd8362f6640 1 librados: shutdown 2024-02-06T21:41:37.072 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-02-06T21:41:37.072 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-06T21:41:37.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.179+0000 7f98ada33640 10 monclient: get_monmap_and_config 2024-02-06T21:41:37.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.179+0000 7f98ada33640 10 monclient: build_initial_monmap 2024-02-06T21:41:37.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.179+0000 7f98ada33640 10 monclient: monmap: 2024-02-06T21:41:37.181 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:37.181 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.182 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:37.181218+0000 2024-02-06T21:41:37.182 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:37.181218+0000 2024-02-06T21:41:37.182 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:37.182 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.182 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:37.182 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:37.182 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:37.182 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:37.182 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:37.182 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.180+0000 7f98ada33640 1 Processor -- start 2024-02-06T21:41:37.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.180+0000 7f98ada33640 1 -- start start 2024-02-06T21:41:37.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.180+0000 7f98ada33640 10 monclient: init 2024-02-06T21:41:37.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98ada33640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:37.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98ada33640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-02-06T21:41:37.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98ada33640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f98a80b54b0 0x7f98a80a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98ada33640 10 monclient(hunting): picked mon.noname-c con 0x7f98a80b54b0 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:37.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98ada33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98a813f830 0x7f98a813fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98ada33640 10 monclient(hunting): picked mon.noname-b con 0x7f98a813f830 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:37.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98ada33640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98a8142680 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:37.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98ada33640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a8143200 con 0x7f98a8142680 2024-02-06T21:41:37.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98ada33640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98ada33640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a80ab520 con 0x7f98a80b54b0 2024-02-06T21:41:37.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98ada33640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98ada33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a80b5890 con 0x7f98a813f830 2024-02-06T21:41:37.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98ada33640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:37.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98ada33640 10 monclient(hunting): authenticate will time out at 1327.518729s 2024-02-06T21:41:37.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98a67fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98a813f830 0x7f98a813fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.181+0000 7f98a6ffd640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f98a80b54b0 0x7f98a80a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.182+0000 7f98a67fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98a813f830 0x7f98a813fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:56754/0 (socket says 172.21.15.120:56754) 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.182+0000 7f98a67fc640 1 -- 172.21.15.120:0/1403053315 learned_addr learned my addr 172.21.15.120:0/1403053315 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.182+0000 7f98a67fc640 10 monclient(hunting): get_auth_request con 0x7f98a813f830 auth_method 0 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.182+0000 7f98a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.182+0000 7f98a67fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.182+0000 7f98a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.182+0000 7f98a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.182+0000 7f98a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.182+0000 7f98a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.182+0000 7f98a6ffd640 10 monclient(hunting): get_auth_request con 0x7f98a80b54b0 auth_method 0 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.182+0000 7f98a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.182+0000 7f98a6ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.182+0000 7f98a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.183+0000 7f98a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:37.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.183+0000 7f98a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:37.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.183+0000 7f98a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:37.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.184+0000 7f98a67fc640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2024-02-06T21:41:37.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.184+0000 7f98a67fc640 1 -- 172.21.15.120:0/1403053315 >> v2:172.21.15.135:3300/0 conn(0x7f98a80b54b0 msgr2=0x7f98a80a9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.184+0000 7f98a67fc640 1 --2- 172.21.15.120:0/1403053315 >> v2:172.21.15.135:3300/0 conn(0x7f98a80b54b0 0x7f98a80a9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.184+0000 7f98a67fc640 1 -- 172.21.15.120:0/1403053315 >> v1:172.21.15.135:6789/0 conn(0x7f98a8142680 legacy=0x7f98a8142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.184+0000 7f98a67fc640 10 monclient: _finish_hunting 0 2024-02-06T21:41:37.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.184+0000 7f98a67fc640 1 monclient: found mon.noname-b 2024-02-06T21:41:37.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.184+0000 7f98a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:37.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.184+0000 7f98a67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:37.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.184+0000 7f98a67fc640 1 -- 172.21.15.120:0/1403053315 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a80ab720 con 0x7f98a813f830 2024-02-06T21:41:37.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.186+0000 7f98a67fc640 1 --2- 172.21.15.120:0/1403053315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98a813f830 0x7f98a813fc70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9898009870 tx=0x7f989802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7080f9e40529a38d server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:37.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98a5ffb640 1 -- 172.21.15.120:0/1403053315 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898031640 con 0x7f98a813f830 2024-02-06T21:41:37.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:37.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:37.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98a5ffb640 10 monclient: dump: 2024-02-06T21:41:37.193 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:37.193 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.194 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.194 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.194 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:37.194 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.194 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:37.194 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:37.194 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:37.194 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98a5ffb640 10 monclient: _finish_auth 0 2024-02-06T21:41:37.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98a5ffb640 10 monclient: _check_auth_tickets 2024-02-06T21:41:37.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:37.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98a5ffb640 1 -- 172.21.15.120:0/1403053315 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98980317a0 con 0x7f98a813f830 2024-02-06T21:41:37.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98ada33640 5 monclient: authenticate success, global_id 4250 2024-02-06T21:41:37.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98ada33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:37.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:37.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98ada33640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:37.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98ada33640 4 set_mon_vals no callback set 2024-02-06T21:41:37.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98a5ffb640 1 -- 172.21.15.120:0/1403053315 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898031b30 con 0x7f98a813f830 2024-02-06T21:41:37.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:37.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:37.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.189+0000 7f98a5ffb640 10 monclient: dump: 2024-02-06T21:41:37.195 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:37.195 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.195 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.195 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.195 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:37.195 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.195 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:37.195 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:37.195 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:37.196 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.190+0000 7f98ada33640 10 monclient: shutdown 2024-02-06T21:41:37.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.190+0000 7f98ada33640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:37.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.190+0000 7f98ada33640 1 -- 172.21.15.120:0/1403053315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98a813f830 msgr2=0x7f98a813fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:37.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.190+0000 7f98ada33640 1 --2- 172.21.15.120:0/1403053315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98a813f830 0x7f98a813fc70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9898009870 tx=0x7f989802b8f0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.190+0000 7f98ada33640 1 -- 172.21.15.120:0/1403053315 shutdown_connections 2024-02-06T21:41:37.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.190+0000 7f98ada33640 1 --2- 172.21.15.120:0/1403053315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98a813f830 0x7f98a813fc70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.190+0000 7f98ada33640 1 --2- 172.21.15.120:0/1403053315 >> v2:172.21.15.135:3300/0 conn(0x7f98a80b54b0 0x7f98a80a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.190+0000 7f98ada33640 1 -- 172.21.15.120:0/1403053315 >> 172.21.15.120:0/1403053315 conn(0x7f98a80b8660 msgr2=0x7f98a80b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:37.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.190+0000 7f98ada33640 1 -- 172.21.15.120:0/1403053315 shutdown_connections 2024-02-06T21:41:37.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.190+0000 7f98ada33640 1 -- 172.21.15.120:0/1403053315 wait complete. 2024-02-06T21:41:37.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.191+0000 7f98ada33640 10 monclient: build_initial_monmap 2024-02-06T21:41:37.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.191+0000 7f98ada33640 10 monclient: monmap: 2024-02-06T21:41:37.196 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:37.197 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:37.197 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:37.197 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:37.197 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:37.197 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.197 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:37.197 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:37.197 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:37.197 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.191+0000 7f98ada33640 1 librados: starting msgr at 2024-02-06T21:41:37.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.191+0000 7f98ada33640 1 librados: starting objecter 2024-02-06T21:41:37.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.191+0000 7f98ada33640 1 Processor -- start 2024-02-06T21:41:37.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.191+0000 7f98ada33640 1 -- start start 2024-02-06T21:41:37.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.191+0000 7f98ada33640 1 librados: setting wanted keys 2024-02-06T21:41:37.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.191+0000 7f98ada33640 1 librados: calling monclient init 2024-02-06T21:41:37.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.191+0000 7f98ada33640 10 monclient: init 2024-02-06T21:41:37.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:37.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T21:41:37.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98a80b54b0 0x7f98a8135030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 10 monclient(hunting): picked mon.noname-c con 0x7f98a80b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:37.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98a813f830 0x7f98a80b7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 10 monclient(hunting): picked mon.noname-b con 0x7f98a813f830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:37.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98a8142680 0x7f98a81388b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 10 monclient(hunting): picked mon.noname-a con 0x7f98a8142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:37.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a81382b0 con 0x7f98a8142680 2024-02-06T21:41:37.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a81366c0 con 0x7f98a813f830 2024-02-06T21:41:37.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a8138510 con 0x7f98a80b54b0 2024-02-06T21:41:37.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:37.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98ada33640 10 monclient(hunting): authenticate will time out at 1327.529945s 2024-02-06T21:41:37.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98a6ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98a80b54b0 0x7f98a8135030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.192+0000 7f98a77fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98a8142680 0x7f98a81388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a67fc640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98a813f830 0x7f98a80b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a6ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98a80b54b0 0x7f98a8135030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:56770/0 (socket says 172.21.15.120:56770) 2024-02-06T21:41:37.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a6ffd640 1 -- 172.21.15.120:0/3399525640 learned_addr learned my addr 172.21.15.120:0/3399525640 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:37.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a6ffd640 10 monclient(hunting): get_auth_request con 0x7f98a80b54b0 auth_method 0 2024-02-06T21:41:37.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a6ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a77fe640 10 monclient(hunting): get_auth_request con 0x7f98a8142680 auth_method 0 2024-02-06T21:41:37.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a77fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a67fc640 10 monclient(hunting): get_auth_request con 0x7f98a813f830 auth_method 0 2024-02-06T21:41:37.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a67fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:37.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:37.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:37.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:37.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:37.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.193+0000 7f98a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:37.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.194+0000 7f98a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:37.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.194+0000 7f98a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:37.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.194+0000 7f98a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:37.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.196+0000 7f98a6ffd640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2024-02-06T21:41:37.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.196+0000 7f98a6ffd640 1 -- 172.21.15.120:0/3399525640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98a813f830 msgr2=0x7f98a80b7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:37.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.196+0000 7f98a6ffd640 1 --2- 172.21.15.120:0/3399525640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98a813f830 0x7f98a80b7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.196+0000 7f98a6ffd640 1 -- 172.21.15.120:0/3399525640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98a8142680 msgr2=0x7f98a81388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:37.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.196+0000 7f98a6ffd640 1 --2- 172.21.15.120:0/3399525640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98a8142680 0x7f98a81388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.196+0000 7f98a6ffd640 10 monclient: _finish_hunting 0 2024-02-06T21:41:37.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.196+0000 7f98a6ffd640 1 monclient: found mon.noname-c 2024-02-06T21:41:37.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.196+0000 7f98a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:37.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.196+0000 7f98a6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:41:37.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.196+0000 7f98a6ffd640 1 -- 172.21.15.120:0/3399525640 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98980094d0 con 0x7f98a80b54b0 2024-02-06T21:41:37.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.198+0000 7f98a6ffd640 1 --2- 172.21.15.120:0/3399525640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98a80b54b0 0x7f98a8135030 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f989c00e9b0 tx=0x7f989c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:37.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.198+0000 7f988ffff640 10 client.?.objecter ms_handle_connect 0x7f98a80b54b0 2024-02-06T21:41:37.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.198+0000 7f988ffff640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:37.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f988ffff640 1 -- 172.21.15.120:0/3399525640 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c009670 con 0x7f98a80b54b0 2024-02-06T21:41:37.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f988ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:37.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f988ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:37.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f988ffff640 10 monclient: dump: 2024-02-06T21:41:37.206 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:37.207 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.207 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.207 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.207 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:37.207 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.207 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:37.207 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:37.207 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:37.207 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f988ffff640 10 monclient: _finish_auth 0 2024-02-06T21:41:37.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f988ffff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:37.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f988ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:37.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f988ffff640 1 -- 172.21.15.120:0/3399525640 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c00c9a0 con 0x7f98a80b54b0 2024-02-06T21:41:37.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f98ada33640 5 monclient: authenticate success, global_id 4253 2024-02-06T21:41:37.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f98ada33640 10 monclient: _renew_subs 2024-02-06T21:41:37.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f98ada33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:37.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f98ada33640 1 -- 172.21.15.120:0/3399525640 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a80b7870 con 0x7f98a80b54b0 2024-02-06T21:41:37.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f988ffff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:37.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f988ffff640 1 -- 172.21.15.120:0/3399525640 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c00cd50 con 0x7f98a80b54b0 2024-02-06T21:41:37.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f988ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:37.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f988ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:37.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f988ffff640 10 monclient: dump: 2024-02-06T21:41:37.208 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:37.208 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.208 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.208 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.208 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:37.208 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.208 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:37.209 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:37.209 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:37.209 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f98a57fa640 4 set_mon_vals no callback set 2024-02-06T21:41:37.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f98ada33640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:37.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f98ada33640 10 monclient: _renew_subs 2024-02-06T21:41:37.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f98ada33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:37.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f98ada33640 1 -- 172.21.15.120:0/3399525640 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a8136dd0 con 0x7f98a80b54b0 2024-02-06T21:41:37.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.202+0000 7f98ada33640 1 librados: init done 2024-02-06T21:41:37.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.203+0000 7f98ada33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:37.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.203+0000 7f98ada33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:37.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.203+0000 7f98ada33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:37.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.203+0000 7f98ada33640 1 -- 172.21.15.120:0/3399525640 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a804ef90 con 0x7f98a80b54b0 2024-02-06T21:41:37.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.446+0000 7f988ffff640 1 -- 172.21.15.120:0/3399525640 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f989c013470 con 0x7f98a80b54b0 2024-02-06T21:41:37.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.447+0000 7f988ffff640 1 --2- 172.21.15.120:0/3399525640 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f989406d1a0 0x7f989406f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.447+0000 7f98a67fc640 1 --2- 172.21.15.120:0/3399525640 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f989406d1a0 0x7f989406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.447+0000 7f98a67fc640 10 monclient: get_auth_request con 0x7f989406d1a0 auth_method 0 2024-02-06T21:41:37.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.448+0000 7f988ffff640 10 client.4253.objecter ms_handle_connect 0x7f989406d1a0 2024-02-06T21:41:37.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.448+0000 7f98a67fc640 1 --2- 172.21.15.120:0/3399525640 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f989406d1a0 0x7f989406f660 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9898009870 tx=0x7f989803c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:37.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.461+0000 7f988ffff640 1 -- 172.21.15.120:0/3399525640 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f989c059230 con 0x7f98a80b54b0 2024-02-06T21:41:37.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.461+0000 7f988ffff640 10 client.4253.objecter ms_dispatch 0x7f98a8000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:37.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.461+0000 7f988ffff640 3 client.4253.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:37.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.461+0000 7f988ffff640 3 client.4253.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:37.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.461+0000 7f988ffff640 20 client.4253.objecter dump_active .. 0 homeless 2024-02-06T21:41:37.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.461+0000 7f988ffff640 1 -- 172.21.15.120:0/3399525640 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f989c0554d0 con 0x7f98a80b54b0 2024-02-06T21:41:37.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.461+0000 7f988ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:37.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.461+0000 7f988ffff640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.679+0000 7f98ada33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.679+0000 7f98ada33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.679+0000 7f98ada33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:37.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.679+0000 7f98ada33640 1 -- 172.21.15.120:0/3399525640 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98a8136dd0 con 0x7f98a80b54b0 2024-02-06T21:41:37.730 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.728+0000 7f988ffff640 1 -- 172.21.15.120:0/3399525640 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14777 (secure 0 0 0) 0x7f98a8136dd0 con 0x7f98a80b54b0 2024-02-06T21:41:37.730 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.728+0000 7f988ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T21:41:37.730 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.728+0000 7f988ffff640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:37.730 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:41:37.730 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":23,"fsid":"4e4791bf-0a02-4478-9517-2d2e893f4f7c","created":"2024-02-06T21:40:31.800075+0000","modified":"2024-02-06T21:41:36.314825+0000","last_up_change":"2024-02-06T21:41:29.190309+0000","last_in_change":"2024-02-06T21:40:37.354288+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-06T21:41:20.093970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-02-06T21:41:33.186378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4bd0f591-2cf1-42da-8ac1-dce3188580c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6817","nonce":2062960776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6819","nonce":2062960776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6823","nonce":2062960776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6821","nonce":2062960776}]},"public_addr":"172.21.15.120:6817/2062960776","cluster_addr":"172.21.15.120:6819/2062960776","heartbeat_back_addr":"172.21.15.120:6823/2062960776","heartbeat_front_addr":"172.21.15.120:6821/2062960776","state":["exists","up"]},{"osd":1,"uuid":"81dc801c-3768-4c2c-9adb-6c8300a341f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6809","nonce":3348315891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6811","nonce":3348315891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6815","nonce":3348315891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6813","nonce":3348315891}]},"public_addr":"172.21.15.120:6809/3348315891","cluster_addr":"172.21.15.120:6811/3348315891","heartbeat_back_addr":"172.21.15.120:6815/3348315891","heartbeat_front_addr":"172.21.15.120:6813/3348315891","state":["exists","up"]},{"osd":2,"uuid":"9289721b-c13f-4a17-be68-3698090f69af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6825","nonce":3229495265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6827","nonce":3229495265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6831","nonce":3229495265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6829","nonce":3229495265}]},"public_addr":"172.21.15.120:6825/3229495265","cluster_addr":"172.21.15.120:6827/3229495265","heartbeat_back_addr":"172.21.15.120:6831/3229495265","heartbeat_front_addr":"172.21.15.120:6829/3229495265","state":["exists","up"]},{"osd":3,"uuid":"53534db0-2d4d-44ed-90e8-d67ea878b51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6801","nonce":3324566974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6803","nonce":3324566974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6807","nonce":3324566974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6805","nonce":3324566974}]},"public_addr":"172.21.15.120:6801/3324566974","cluster_addr":"172.21.15.120:6803/3324566974","heartbeat_back_addr":"172.21.15.120:6807/3324566974","heartbeat_front_addr":"172.21.15.120:6805/3324566974","state":["exists","up"]},{"osd":4,"uuid":"e4205ab5-787c-44e0-9b96-273b8c9cf1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6803","nonce":2191547918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6805","nonce":2191547918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6809","nonce":2191547918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6807","nonce":2191547918}]},"public_addr":"172.21.15.135:6803/2191547918","cluster_addr":"172.21.15.135:6805/2191547918","heartbeat_back_addr":"172.21.15.135:6809/2191547918","heartbeat_front_addr":"172.21.15.135:6807/2191547918","state":["exists","up"]},{"osd":5,"uuid":"43b600bc-3afe-44bd-bd3f-690106e7e0ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6819","nonce":563193066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6821","nonce":563193066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6826","nonce":563193066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6823","nonce":563193066}]},"public_addr":"172.21.15.135:6819/563193066","cluster_addr":"172.21.15.135:6821/563193066","heartbeat_back_addr":"172.21.15.135:6826/563193066","heartbeat_front_addr":"172.21.15.135:6823/563193066","state":["exists","up"]},{"osd":6,"uuid":"1d6b9f96-1f9d-44f2-83b0-93f5c342fafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6811","nonce":1080459988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6813","nonce":1080459988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6817","nonce":1080459988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6815","nonce":1080459988}]},"public_addr":"172.21.15.135:6811/1080459988","cluster_addr":"172.21.15.135:6813/1080459988","heartbeat_back_addr":"172.21.15.135:6817/1080459988","heartbeat_front_addr":"172.21.15.135:6815/1080459988","state":["exists","up"]},{"osd":7,"uuid":"ecae00f5-8e29-47dd-90d5-e51503bc9dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6825","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6827","nonce":3740758337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6829","nonce":3740758337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6833","nonce":3740758337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6831","nonce":3740758337}]},"public_addr":"172.21.15.135:6827/3740758337","cluster_addr":"172.21.15.135:6829/3740758337","heartbeat_back_addr":"172.21.15.135:6833/3740758337","heartbeat_front_addr":"172.21.15.135:6831/3740758337","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.344503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:17.073640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.245112+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.202999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:15.825107+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:15.918704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.203923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:15.753628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-06T21:41:37.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 10 librados: watch_flush enter 2024-02-06T21:41:37.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 10 librados: watch_flush exit 2024-02-06T21:41:37.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 20 client.4253.objecter shutdown clearing up homeless session... 2024-02-06T21:41:37.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 10 client.4253.objecter successfully canceled tick 2024-02-06T21:41:37.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 1 -- 172.21.15.120:0/3399525640 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f989406d1a0 msgr2=0x7f989406f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:37.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 1 --2- 172.21.15.120:0/3399525640 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f989406d1a0 0x7f989406f660 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9898009870 tx=0x7f989803c040 comp rx=0 tx=0).stop 2024-02-06T21:41:37.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 10 monclient: shutdown 2024-02-06T21:41:37.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:37.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 1 -- 172.21.15.120:0/3399525640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98a80b54b0 msgr2=0x7f98a8135030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:37.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 1 --2- 172.21.15.120:0/3399525640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98a80b54b0 0x7f98a8135030 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f989c00e9b0 tx=0x7f989c00ee70 comp rx=0 tx=0).stop 2024-02-06T21:41:37.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 1 -- 172.21.15.120:0/3399525640 shutdown_connections 2024-02-06T21:41:37.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 1 --2- 172.21.15.120:0/3399525640 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f989406d1a0 0x7f989406f660 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 1 --2- 172.21.15.120:0/3399525640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98a813f830 0x7f98a80b7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 1 --2- 172.21.15.120:0/3399525640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98a8142680 0x7f98a81388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 1 --2- 172.21.15.120:0/3399525640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98a80b54b0 0x7f98a8135030 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 1 -- 172.21.15.120:0/3399525640 >> 172.21.15.120:0/3399525640 conn(0x7f98a80b8660 msgr2=0x7f98a813bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:37.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 1 -- 172.21.15.120:0/3399525640 shutdown_connections 2024-02-06T21:41:37.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 1 -- 172.21.15.120:0/3399525640 wait complete. 2024-02-06T21:41:37.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.730+0000 7f98ada33640 1 librados: shutdown 2024-02-06T21:41:37.744 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-02-06T21:41:37.744 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-02-06T21:41:37.745 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-02-06T21:41:37.745 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-02-06T21:41:37.745 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-02-06T21:41:37.746 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-02-06T21:41:37.746 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-02-06T21:41:37.747 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-02-06T21:41:37.914 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.912+0000 7f578304a640 10 monclient: get_monmap_and_config 2024-02-06T21:41:37.914 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.912+0000 7f578304a640 10 monclient: build_initial_monmap 2024-02-06T21:41:37.914 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.912+0000 7f578304a640 10 monclient: monmap: 2024-02-06T21:41:37.914 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:37.914 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.914 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:37.913519+0000 2024-02-06T21:41:37.914 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:37.913519+0000 2024-02-06T21:41:37.914 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:37.915 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.915 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:37.915 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:37.915 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:37.915 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:37.915 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:37.915 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.916 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.913+0000 7f578304a640 1 Processor -- start 2024-02-06T21:41:37.916 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.913+0000 7f578304a640 1 -- start start 2024-02-06T21:41:37.916 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.913+0000 7f578304a640 10 monclient: init 2024-02-06T21:41:37.916 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.912+0000 7fdf2bc63640 10 monclient: get_monmap_and_config 2024-02-06T21:41:37.916 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.912+0000 7fdf2bc63640 10 monclient: build_initial_monmap 2024-02-06T21:41:37.916 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.913+0000 7fdf2bc63640 10 monclient: monmap: 2024-02-06T21:41:37.916 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:37.916 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.916 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:37.914270+0000 2024-02-06T21:41:37.916 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:37.914270+0000 2024-02-06T21:41:37.916 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:37.917 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.917 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:37.917 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:37.917 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:37.917 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:37.917 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:37.917 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.918 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.915+0000 7fdf2bc63640 1 Processor -- start 2024-02-06T21:41:37.918 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.915+0000 7fdf2bc63640 1 -- start start 2024-02-06T21:41:37.918 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.916+0000 7fdf2bc63640 10 monclient: init 2024-02-06T21:41:37.918 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.915+0000 7f288359e640 10 monclient: get_monmap_and_config 2024-02-06T21:41:37.918 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.915+0000 7f288359e640 10 monclient: build_initial_monmap 2024-02-06T21:41:37.918 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.915+0000 7f288359e640 10 monclient: monmap: 2024-02-06T21:41:37.918 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:37.918 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.918 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:37.916956+0000 2024-02-06T21:41:37.919 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:37.916956+0000 2024-02-06T21:41:37.919 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:37.919 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.919 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:37.919 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:37.919 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:37.919 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:37.919 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:37.919 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.920 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7fdf2bc63640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:37.920 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7fdf2bc63640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-02-06T21:41:37.920 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7fdf2bc63640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdf2406f440 0x7fdf2406f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.920 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7fdf2bc63640 10 monclient(hunting): picked mon.noname-a con 0x7fdf2406f440 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:37.921 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7fdf2bc63640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf2413ba80 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:37.921 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7fdf2bc63640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf24072010 0x7fdf24148ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.921 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7fdf2bc63640 10 monclient(hunting): picked mon.noname-b con 0x7fdf24072010 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:37.921 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7fdf2bc63640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2413f580 con 0x7fdf2413ba80 2024-02-06T21:41:37.921 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.918+0000 7fdf2bc63640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.921 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.918+0000 7fdf2bc63640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf2406ff70 con 0x7fdf2406f440 2024-02-06T21:41:37.921 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.918+0000 7fdf2bc63640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.921 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.916+0000 7f288359e640 1 Processor -- start 2024-02-06T21:41:37.922 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.918+0000 7fdf2bc63640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf24072cf0 con 0x7fdf24072010 2024-02-06T21:41:37.922 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.918+0000 7fdf2bc63640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:37.922 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.918+0000 7fdf2bc63640 10 monclient(hunting): authenticate will time out at 1328.255742s 2024-02-06T21:41:37.922 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.916+0000 7f288359e640 1 -- start start 2024-02-06T21:41:37.922 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.914+0000 7f578304a640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:37.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.914+0000 7f578304a640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-02-06T21:41:37.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.914+0000 7f578304a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f577c06e780 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:37.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.914+0000 7f578304a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f577c06f8e0 0x7f577c06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.914+0000 7f578304a640 10 monclient(hunting): picked mon.noname-b con 0x7f577c06f8e0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:37.924 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.916+0000 7f288359e640 10 monclient: init 2024-02-06T21:41:37.924 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f288359e640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:37.924 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f288359e640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-02-06T21:41:37.924 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f288359e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f287c02e1f0 0x7f287c02e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.924 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f288359e640 10 monclient(hunting): picked mon.noname-a con 0x7f287c02e1f0 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:37.924 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.914+0000 7f578304a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f577c0710d0 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:37.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.914+0000 7f578304a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f577c149fa0 con 0x7f577c06e780 2024-02-06T21:41:37.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.914+0000 7f578304a640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f577c06dfb0 con 0x7f577c0710d0 2024-02-06T21:41:37.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.914+0000 7f578304a640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.914+0000 7f578304a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577c144430 con 0x7f577c06f8e0 2024-02-06T21:41:37.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.914+0000 7f578304a640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:37.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.914+0000 7f578304a640 10 monclient(hunting): authenticate will time out at 1328.251927s 2024-02-06T21:41:37.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.915+0000 7f5780de6640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f577c06e780 0x7f577c143d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:52980/0 (socket says 172.21.15.120:52980) 2024-02-06T21:41:37.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.915+0000 7f5780de6640 1 -- 172.21.15.120:0/1468167761 learned_addr learned my addr 172.21.15.120:0/1468167761 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:41:37.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.915+0000 7f577bfff640 1 --2- 172.21.15.120:0/1468167761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f577c06f8e0 0x7f577c06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.915+0000 7f577bfff640 10 monclient(hunting): get_auth_request con 0x7f577c06f8e0 auth_method 0 2024-02-06T21:41:37.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.915+0000 7f577bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.915+0000 7f577bfff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.915+0000 7f577bfff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.926 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f577bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:37.926 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f577bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:37.926 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f577bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:37.928 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f288359e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f287c032960 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:37.928 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f288359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f287c0cfc80 0x7f287c0d6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.928 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f288359e640 10 monclient(hunting): picked mon.noname-b con 0x7f287c0cfc80 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:37.928 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f288359e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f287c02f8f0 con 0x7f287c032960 2024-02-06T21:41:37.928 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f288359e640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.928 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f288359e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f287c02fbd0 con 0x7f287c02e1f0 2024-02-06T21:41:37.929 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f288359e640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.929 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f288359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287c02d710 con 0x7f287c0cfc80 2024-02-06T21:41:37.929 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f288359e640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:37.929 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.917+0000 7f288359e640 10 monclient(hunting): authenticate will time out at 1328.255449s 2024-02-06T21:41:37.930 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f577b7fe640 1 -- 172.21.15.120:0/1468167761 <== mon.0 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659418727 0 0) 0x7f577c149fa0 con 0x7f577c06e780 2024-02-06T21:41:37.930 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f577b7fe640 10 monclient(hunting): _negotiate 2024-02-06T21:41:37.930 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f577b7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.930 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f577b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.930 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f577b7fe640 10 monclient(hunting): my global_id is 4284 2024-02-06T21:41:37.930 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f577b7fe640 1 -- 172.21.15.120:0/1468167761 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5764003520 con 0x7f577c06e780 2024-02-06T21:41:37.930 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f0ba327b640 10 monclient: get_monmap_and_config 2024-02-06T21:41:37.930 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f0ba327b640 10 monclient: build_initial_monmap 2024-02-06T21:41:37.930 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f0ba327b640 10 monclient: monmap: 2024-02-06T21:41:37.931 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:37.931 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.931 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:37.920368+0000 2024-02-06T21:41:37.931 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:37.920368+0000 2024-02-06T21:41:37.931 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:37.931 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.932 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:37.932 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:37.932 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:37.932 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:37.932 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:37.932 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.932 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f577bfff640 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2024-02-06T21:41:37.932 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f577bfff640 1 -- 172.21.15.120:0/1468167761 >> v1:172.21.15.135:6789/0 conn(0x7f577c0710d0 legacy=0x7f577c14a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.932 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f577bfff640 1 -- 172.21.15.120:0/1468167761 >> v1:172.21.15.120:6789/0 conn(0x7f577c06e780 legacy=0x7f577c143d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:37.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f577bfff640 10 monclient: _finish_hunting 0 2024-02-06T21:41:37.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f577bfff640 1 monclient: found mon.noname-b 2024-02-06T21:41:37.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f577bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:37.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f577bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:37.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.919+0000 7f577bfff640 1 -- 172.21.15.120:0/1468167761 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f577c144570 con 0x7f577c06f8e0 2024-02-06T21:41:37.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.920+0000 7f0ba327b640 1 Processor -- start 2024-02-06T21:41:37.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.920+0000 7f0ba327b640 1 -- start start 2024-02-06T21:41:37.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.920+0000 7f0ba327b640 10 monclient: init 2024-02-06T21:41:37.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f0ba327b640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:37.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f0ba327b640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-02-06T21:41:37.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f0ba327b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0b9c06b590 0x7f0b9c13ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f0ba327b640 10 monclient(hunting): picked mon.noname-a con 0x7f0b9c06b590 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:37.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f0ba327b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b9c0a9ce0 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:37.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f0ba327b640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0b9c06ca30 0x7f0b9c13f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f0ba327b640 10 monclient(hunting): picked mon.noname-c con 0x7f0b9c06ca30 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:37.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f0ba327b640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9c0aa290 con 0x7f0b9c0a9ce0 2024-02-06T21:41:37.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f0ba327b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f0ba327b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9c13f690 con 0x7f0b9c06b590 2024-02-06T21:41:37.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f0ba327b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f0ba327b640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9c06b970 con 0x7f0b9c06ca30 2024-02-06T21:41:37.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f0ba327b640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:37.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f0ba327b640 10 monclient(hunting): authenticate will time out at 1328.258909s 2024-02-06T21:41:37.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7fdf2a200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf24072010 0x7fdf24148ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f57a473d640 10 monclient: get_monmap_and_config 2024-02-06T21:41:37.936 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f57a473d640 10 monclient: build_initial_monmap 2024-02-06T21:41:37.936 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7f57a473d640 10 monclient: monmap: 2024-02-06T21:41:37.936 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:37.936 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.936 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:37.922994+0000 2024-02-06T21:41:37.936 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:37.922994+0000 2024-02-06T21:41:37.936 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:37.936 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.936 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:37.936 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:37.936 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:37.936 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:37.937 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:37.937 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.937 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7fdf2a200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf24072010 0x7fdf24148ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:56790/0 (socket says 172.21.15.120:56790) 2024-02-06T21:41:37.937 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7fdf2a200640 1 -- 172.21.15.120:0/849898423 learned_addr learned my addr 172.21.15.120:0/849898423 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:37.937 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7fdf2a200640 10 monclient(hunting): get_auth_request con 0x7fdf24072010 auth_method 0 2024-02-06T21:41:37.937 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7fdf2a200640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.937 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7fdf2a200640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.937 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.921+0000 7fdf2a200640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.937 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.922+0000 7fdf299ff640 1 --2- 172.21.15.120:0/849898423 >> v2:172.21.15.120:3300/0 conn(0x7fdf2406f440 0x7fdf2406f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.938 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.922+0000 7f038c245640 10 monclient: get_monmap_and_config 2024-02-06T21:41:37.938 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.922+0000 7f038c245640 10 monclient: build_initial_monmap 2024-02-06T21:41:37.938 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.922+0000 7f038c245640 10 monclient: monmap: 2024-02-06T21:41:37.939 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:37.939 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.939 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:37.923850+0000 2024-02-06T21:41:37.939 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:37.923850+0000 2024-02-06T21:41:37.939 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:37.939 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.939 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:37.939 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:37.939 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:37.939 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:37.939 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:37.939 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.922+0000 7f57a473d640 1 Processor -- start 2024-02-06T21:41:37.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.922+0000 7f57a473d640 1 -- start start 2024-02-06T21:41:37.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.922+0000 7f57a473d640 10 monclient: init 2024-02-06T21:41:37.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.922+0000 7f0ba1017640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0b9c06b590 0x7f0b9c13ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f1540899640 10 monclient: get_monmap_and_config 2024-02-06T21:41:37.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f1540899640 10 monclient: build_initial_monmap 2024-02-06T21:41:37.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f1540899640 10 monclient: monmap: 2024-02-06T21:41:37.941 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:37.941 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.941 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:37.924480+0000 2024-02-06T21:41:37.941 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:37.924480+0000 2024-02-06T21:41:37.941 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:37.941 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.941 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:37.941 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:37.941 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:37.941 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:37.942 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:37.942 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.942 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7fdf2a200640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:37.942 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7fdf2a200640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:37.942 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7fdf2a200640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:37.942 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7fdf299ff640 10 monclient(hunting): get_auth_request con 0x7fdf2406f440 auth_method 0 2024-02-06T21:41:37.942 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7fdf299ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.942 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7fdf299ff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.942 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7fdf299ff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.943 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f038c245640 1 Processor -- start 2024-02-06T21:41:37.943 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f0ba1017640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0b9c06b590 0x7f0b9c13ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60106/0 (socket says 172.21.15.120:60106) 2024-02-06T21:41:37.943 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f0ba1017640 1 -- 172.21.15.120:0/466123236 learned_addr learned my addr 172.21.15.120:0/466123236 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:37.943 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f0ba1017640 10 monclient(hunting): get_auth_request con 0x7f0b9c06b590 auth_method 0 2024-02-06T21:41:37.943 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f0ba1017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.943 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f0ba1017640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.943 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f0ba1017640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.944 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f038c245640 1 -- start start 2024-02-06T21:41:37.944 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f038c245640 10 monclient: init 2024-02-06T21:41:37.944 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f57a473d640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:37.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f57a473d640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-02-06T21:41:37.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f57a473d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5794007440 0x7f579400a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f57a473d640 10 monclient(hunting): picked mon.noname-b con 0x7f5794007440 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:37.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f57a473d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57940c8f60 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:37.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f57a473d640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f57940cf6e0 0x7f57940cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f57a473d640 10 monclient(hunting): picked mon.noname-c con 0x7f57940cf6e0 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:37.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f57a473d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579400b080 con 0x7f57940c8f60 2024-02-06T21:41:37.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f57a473d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f57a473d640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57940c9ab0 con 0x7f57940cf6e0 2024-02-06T21:41:37.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f57a473d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f57a473d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5794007820 con 0x7f5794007440 2024-02-06T21:41:37.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f57a473d640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:37.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.923+0000 7f57a473d640 10 monclient(hunting): authenticate will time out at 1328.261508s 2024-02-06T21:41:37.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f1540899640 1 Processor -- start 2024-02-06T21:41:37.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f0ba1017640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:37.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f0ba1017640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:37.947 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f0ba1017640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:37.947 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f288259c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f287c02e1f0 0x7f287c02e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.947 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f038c245640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:37.947 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f038c245640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-02-06T21:41:37.947 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f038c245640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c007190 0x7f037c0c5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.947 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f038c245640 10 monclient(hunting): picked mon.noname-b con 0x7f037c007190 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:37.947 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f038c245640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f037c007570 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:37.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f038c245640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f037c009030 0x7f037c0d2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f038c245640 10 monclient(hunting): picked mon.noname-a con 0x7f037c009030 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:37.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f038c245640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f037c0304b0 con 0x7f037c007570 2024-02-06T21:41:37.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f038c245640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f038c245640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f037c0c7cc0 con 0x7f037c009030 2024-02-06T21:41:37.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f038c245640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f038c245640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f037c030700 con 0x7f037c007190 2024-02-06T21:41:37.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f038c245640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:37.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f038c245640 10 monclient(hunting): authenticate will time out at 1328.262089s 2024-02-06T21:41:37.949 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f1540899640 1 -- start start 2024-02-06T21:41:37.949 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f1540899640 10 monclient: init 2024-02-06T21:41:37.950 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f288259c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f287c02e1f0 0x7f287c02e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60092/0 (socket says 172.21.15.120:60092) 2024-02-06T21:41:37.950 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f288259c640 1 -- 172.21.15.120:0/3752993568 learned_addr learned my addr 172.21.15.120:0/3752993568 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:37.950 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f288259c640 10 monclient(hunting): get_auth_request con 0x7f287c02e1f0 auth_method 0 2024-02-06T21:41:37.950 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f288259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.950 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f288259c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.950 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.924+0000 7f288259c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.951 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7f1540899640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:37.951 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7f1540899640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-02-06T21:41:37.951 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7f1540899640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f153c0b5170 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:37.952 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7f1540899640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f153c13ff70 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:37.952 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7f1540899640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f153c0a97c0 0x7f153c142d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.952 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7f1540899640 10 monclient(hunting): picked mon.noname-c con 0x7f153c0a97c0 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:37.952 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7f1540899640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c0aafb0 con 0x7f153c13ff70 2024-02-06T21:41:37.952 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7f1540899640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c0ab290 con 0x7f153c0b5170 2024-02-06T21:41:37.952 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7f1540899640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.952 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7f1540899640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153c0b5550 con 0x7f153c0a97c0 2024-02-06T21:41:37.952 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7f1540899640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:37.952 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7f1540899640 10 monclient(hunting): authenticate will time out at 1328.263130s 2024-02-06T21:41:37.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7fdf2a200640 10 monclient(hunting): handle_auth_done global_id 4259 payload 274 2024-02-06T21:41:37.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7fdf2a200640 1 -- 172.21.15.120:0/849898423 >> v2:172.21.15.120:3300/0 conn(0x7fdf2406f440 msgr2=0x7fdf2406f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7fdf2a200640 1 --2- 172.21.15.120:0/849898423 >> v2:172.21.15.120:3300/0 conn(0x7fdf2406f440 0x7fdf2406f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7fdf2a200640 1 -- 172.21.15.120:0/849898423 >> v1:172.21.15.135:6789/0 conn(0x7fdf2413ba80 legacy=0x7fdf24071900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7fdf2a200640 10 monclient: _finish_hunting 0 2024-02-06T21:41:37.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7fdf2a200640 1 monclient: found mon.noname-b 2024-02-06T21:41:37.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7fdf2a200640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:37.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7fdf2a200640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:37.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7fdf2a200640 1 -- 172.21.15.120:0/849898423 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2406e960 con 0x7fdf24072010 2024-02-06T21:41:37.954 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7fd6e91bc640 10 monclient: get_monmap_and_config 2024-02-06T21:41:37.954 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.925+0000 7fd6e91bc640 10 monclient: build_initial_monmap 2024-02-06T21:41:37.954 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.926+0000 7fd6e91bc640 10 monclient: monmap: 2024-02-06T21:41:37.954 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:37.954 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.954 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:37.927272+0000 2024-02-06T21:41:37.954 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:37.927272+0000 2024-02-06T21:41:37.954 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:37.954 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.955 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:37.955 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:37.955 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:37.955 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:37.955 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:37.955 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.926+0000 7f0ba1017640 10 monclient(hunting): handle_auth_done global_id 4287 payload 274 2024-02-06T21:41:37.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.926+0000 7f0ba1017640 1 -- 172.21.15.120:0/466123236 >> v2:172.21.15.135:3300/0 conn(0x7f0b9c06ca30 msgr2=0x7f0b9c13f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.926+0000 7f0ba1017640 1 --2- 172.21.15.120:0/466123236 >> v2:172.21.15.135:3300/0 conn(0x7f0b9c06ca30 0x7f0b9c13f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.926+0000 7f0ba1017640 1 -- 172.21.15.120:0/466123236 >> v1:172.21.15.135:6789/0 conn(0x7f0b9c0a9ce0 legacy=0x7f0b9c0acbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.926+0000 7f0ba1017640 10 monclient: _finish_hunting 0 2024-02-06T21:41:37.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.926+0000 7f0ba1017640 1 monclient: found mon.noname-a 2024-02-06T21:41:37.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.926+0000 7f0ba1017640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:37.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.926+0000 7f0ba1017640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:37.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.926+0000 7f0ba1017640 1 -- 172.21.15.120:0/466123236 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c0abec0 con 0x7f0b9c06b590 2024-02-06T21:41:37.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.926+0000 7f2882d9d640 1 --2- 172.21.15.120:0/3752993568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f287c0cfc80 0x7f287c0d6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.926+0000 7fd6e91bc640 1 Processor -- start 2024-02-06T21:41:37.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.926+0000 7fd6e91bc640 1 -- start start 2024-02-06T21:41:37.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.926+0000 7fd6e91bc640 10 monclient: init 2024-02-06T21:41:37.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.927+0000 7fd6e91bc640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:37.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.927+0000 7fd6e91bc640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-02-06T21:41:37.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.927+0000 7fd6e91bc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6dc006fd0 0x7fd6dc0cb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.927+0000 7fd6e91bc640 10 monclient(hunting): picked mon.noname-b con 0x7fd6dc006fd0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:37.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.927+0000 7fd6e91bc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6dc0073b0 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:37.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.927+0000 7fd6e91bc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd6dc033930 0x7fd6dc0d24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.927+0000 7fd6e91bc640 10 monclient(hunting): picked mon.noname-a con 0x7fd6dc033930 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:37.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.927+0000 7fd6e91bc640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6dc0357e0 con 0x7fd6dc0073b0 2024-02-06T21:41:37.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.927+0000 7fd6e91bc640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.927+0000 7fd6e91bc640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6dc0c5890 con 0x7fd6dc033930 2024-02-06T21:41:37.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.927+0000 7fd6e91bc640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.927+0000 7fd6e91bc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc036660 con 0x7fd6dc006fd0 2024-02-06T21:41:37.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.927+0000 7fd6e91bc640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:37.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.927+0000 7fd6e91bc640 10 monclient(hunting): authenticate will time out at 1328.265156s 2024-02-06T21:41:37.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.928+0000 7f2882d9d640 10 monclient(hunting): get_auth_request con 0x7f287c0cfc80 auth_method 0 2024-02-06T21:41:37.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.928+0000 7f2882d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.928+0000 7f2882d9d640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.928+0000 7f2882d9d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.928+0000 7fdf2a200640 1 --2- 172.21.15.120:0/849898423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf24072010 0x7fdf24148ee0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00d370 tx=0x7fdf1c02e260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afad5ae97da48563 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:37.960 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.929+0000 7f288259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:37.960 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.929+0000 7f288259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:37.960 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.929+0000 7f288259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:37.960 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.929+0000 7f2882d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:37.960 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.929+0000 7f2882d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:37.961 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.929+0000 7f2882d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:37.961 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.930+0000 7f577bfff640 1 --2- 172.21.15.120:0/1468167761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f577c06f8e0 0x7f577c06eb60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f576c0097d0 tx=0x7f576c02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b47e6b58dd8e4c8 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:37.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.932+0000 7f288259c640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-02-06T21:41:37.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.932+0000 7f288259c640 1 -- 172.21.15.120:0/3752993568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f287c0cfc80 msgr2=0x7f287c0d6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.932+0000 7f288259c640 1 --2- 172.21.15.120:0/3752993568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f287c0cfc80 0x7f287c0d6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.932+0000 7f288259c640 1 -- 172.21.15.120:0/3752993568 >> v1:172.21.15.135:6789/0 conn(0x7f287c032960 legacy=0x7f287c0cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.932+0000 7f288259c640 10 monclient: _finish_hunting 0 2024-02-06T21:41:37.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.932+0000 7f288259c640 1 monclient: found mon.noname-a 2024-02-06T21:41:37.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.932+0000 7f288259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:37.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.932+0000 7f288259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:37.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.932+0000 7f288259c640 1 -- 172.21.15.120:0/3752993568 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287c02d850 con 0x7f287c02e1f0 2024-02-06T21:41:37.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.932+0000 7f03897e0640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f037c007570 0x7f037c0c81c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.120:58816/0 (socket says 172.21.15.120:58816) 2024-02-06T21:41:37.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.932+0000 7f03897e0640 1 -- 172.21.15.120:0/3770364555 learned_addr learned my addr 172.21.15.120:0/3770364555 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:41:37.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.934+0000 7f153b7fe640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f153c0b5170 0x7f153c13f860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.120:58830/0 (socket says 172.21.15.120:58830) 2024-02-06T21:41:37.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.934+0000 7f153b7fe640 1 -- 172.21.15.120:0/800412535 learned_addr learned my addr 172.21.15.120:0/800412535 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:41:37.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.937+0000 7f57a2cda640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f57940cf6e0 0x7f57940cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.938+0000 7f57a2cda640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f57940cf6e0 0x7f57940cfb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38534/0 (socket says 172.21.15.120:38534) 2024-02-06T21:41:37.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.938+0000 7f57a2cda640 1 -- 172.21.15.120:0/3694630743 learned_addr learned my addr 172.21.15.120:0/3694630743 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:37.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.938+0000 7f57a2cda640 10 monclient(hunting): get_auth_request con 0x7f57940cf6e0 auth_method 0 2024-02-06T21:41:37.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.938+0000 7f57a2cda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.938+0000 7f57a2cda640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.938+0000 7f57a2cda640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.938+0000 7f0388fdf640 1 -- 172.21.15.120:0/3770364555 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668641289 0 0) 0x7f037c0304b0 con 0x7f037c007570 2024-02-06T21:41:37.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.938+0000 7f0388fdf640 10 monclient(hunting): _negotiate 2024-02-06T21:41:37.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.938+0000 7f0388fdf640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.938+0000 7f0388fdf640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.938+0000 7f0388fdf640 10 monclient(hunting): my global_id is 4285 2024-02-06T21:41:37.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.938+0000 7f0388fdf640 1 -- 172.21.15.120:0/3770364555 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0370003520 con 0x7f037c007570 2024-02-06T21:41:37.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.939+0000 7f153bfff640 1 --2- 172.21.15.120:0/800412535 >> v2:172.21.15.135:3300/0 conn(0x7f153c0a97c0 0x7f153c142d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.939+0000 7fdf289fd640 1 -- 172.21.15.120:0/849898423 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c038600 con 0x7fdf24072010 2024-02-06T21:41:37.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.939+0000 7fdf289fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:37.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.939+0000 7fdf289fd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:37.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.939+0000 7fdf289fd640 10 monclient: dump: 2024-02-06T21:41:37.966 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:37.966 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.966 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.966 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.966 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:37.966 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.966 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:37.966 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:37.966 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:37.966 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.939+0000 7fdf289fd640 10 monclient: _finish_auth 0 2024-02-06T21:41:37.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.939+0000 7fdf289fd640 10 monclient: _check_auth_tickets 2024-02-06T21:41:37.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.939+0000 7fdf289fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:37.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.939+0000 7fdf2bc63640 5 monclient: authenticate success, global_id 4259 2024-02-06T21:41:37.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.939+0000 7fdf2bc63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:37.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.939+0000 7fdf289fd640 1 -- 172.21.15.120:0/849898423 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1c032070 con 0x7fdf24072010 2024-02-06T21:41:37.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.939+0000 7fdf289fd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:37.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.939+0000 7fdf2bc63640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:37.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.939+0000 7fdf2bc63640 4 set_mon_vals no callback set 2024-02-06T21:41:37.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7fdf289fd640 1 -- 172.21.15.120:0/849898423 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c038980 con 0x7fdf24072010 2024-02-06T21:41:37.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7fdf289fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:37.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7fdf289fd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:37.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7fdf289fd640 10 monclient: dump: 2024-02-06T21:41:37.968 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:37.968 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.968 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.968 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.968 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:37.968 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.968 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:37.968 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:37.968 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:37.968 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.969 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7f57a24d9640 1 --2- 172.21.15.120:0/3694630743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5794007440 0x7f579400a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.969 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7f57a2cda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:37.969 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7f57a2cda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:37.969 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7f57a2cda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:37.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7fdf2bc63640 10 monclient: shutdown 2024-02-06T21:41:37.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7fdf2bc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:37.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7fdf2bc63640 1 -- 172.21.15.120:0/849898423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf24072010 msgr2=0x7fdf24148ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:37.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7fdf2bc63640 1 --2- 172.21.15.120:0/849898423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf24072010 0x7fdf24148ee0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00d370 tx=0x7fdf1c02e260 comp rx=0 tx=0).stop 2024-02-06T21:41:37.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7fdf2bc63640 1 -- 172.21.15.120:0/849898423 shutdown_connections 2024-02-06T21:41:37.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7fdf2bc63640 1 --2- 172.21.15.120:0/849898423 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf24072010 0x7fdf24148ee0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7fdf2bc63640 1 --2- 172.21.15.120:0/849898423 >> v2:172.21.15.120:3300/0 conn(0x7fdf2406f440 0x7fdf2406f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7fdf2bc63640 1 -- 172.21.15.120:0/849898423 >> 172.21.15.120:0/849898423 conn(0x7fdf240b07a0 msgr2=0x7fdf240b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:37.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7f153bfff640 10 monclient(hunting): get_auth_request con 0x7f153c0a97c0 auth_method 0 2024-02-06T21:41:37.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7f153bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7f153bfff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7f153bfff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7fdf2bc63640 1 -- 172.21.15.120:0/849898423 shutdown_connections 2024-02-06T21:41:37.972 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7f57a24d9640 10 monclient(hunting): get_auth_request con 0x7f5794007440 auth_method 0 2024-02-06T21:41:37.972 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7f57a24d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.972 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7f57a24d9640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.972 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7f57a24d9640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.973 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.940+0000 7fdf2bc63640 1 -- 172.21.15.120:0/849898423 wait complete. 2024-02-06T21:41:37.973 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.941+0000 7fdf2bc63640 10 monclient: build_initial_monmap 2024-02-06T21:41:37.973 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.941+0000 7fdf2bc63640 10 monclient: monmap: 2024-02-06T21:41:37.973 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:37.973 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:37.973 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:37.973 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:37.973 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:37.973 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.973 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:37.973 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:37.974 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:37.974 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.974 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.941+0000 7fdf2bc63640 1 librados: starting msgr at 2024-02-06T21:41:37.974 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.941+0000 7fdf2bc63640 1 librados: starting objecter 2024-02-06T21:41:37.974 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 1 Processor -- start 2024-02-06T21:41:37.974 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 1 -- start start 2024-02-06T21:41:37.974 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 1 librados: setting wanted keys 2024-02-06T21:41:37.974 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 1 librados: calling monclient init 2024-02-06T21:41:37.974 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 10 monclient: init 2024-02-06T21:41:37.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7f57a2cda640 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2024-02-06T21:41:37.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7f57a2cda640 1 -- 172.21.15.120:0/3694630743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5794007440 msgr2=0x7f579400a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7f57a2cda640 1 --2- 172.21.15.120:0/3694630743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5794007440 0x7f579400a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7f57a2cda640 1 -- 172.21.15.120:0/3694630743 >> v1:172.21.15.120:6789/0 conn(0x7f57940c8f60 legacy=0x7f57940c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7f57a2cda640 10 monclient: _finish_hunting 0 2024-02-06T21:41:37.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fd6e3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6dc006fd0 0x7fd6dc0cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.976 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7f57a2cda640 1 monclient: found mon.noname-c 2024-02-06T21:41:37.976 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7f57a2cda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:37.976 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7f57a2cda640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T21:41:37.976 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7f57a2cda640 1 -- 172.21.15.120:0/3694630743 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579400b360 con 0x7f57940cf6e0 2024-02-06T21:41:37.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:37.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T21:41:37.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf2406f440 0x7fdf240acab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 10 monclient(hunting): picked mon.noname-b con 0x7fdf2406f440 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:37.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf24072010 0x7fdf241d3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 10 monclient(hunting): picked mon.noname-a con 0x7fdf24072010 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:37.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf2413ba80 0x7fdf241d3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 10 monclient(hunting): picked mon.noname-c con 0x7fdf2413ba80 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:37.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2411bf70 con 0x7fdf24072010 2024-02-06T21:41:37.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf240ade70 con 0x7fdf2406f440 2024-02-06T21:41:37.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf24139a00 con 0x7fdf2413ba80 2024-02-06T21:41:37.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:37.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.942+0000 7fdf2bc63640 10 monclient(hunting): authenticate will time out at 1328.280357s 2024-02-06T21:41:37.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7fd6e3fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6dc006fd0 0x7fd6dc0cb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:56836/0 (socket says 172.21.15.120:56836) 2024-02-06T21:41:37.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7fd6e3fff640 1 -- 172.21.15.120:0/3939976003 learned_addr learned my addr 172.21.15.120:0/3939976003 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:37.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7fd6e3fff640 10 monclient(hunting): get_auth_request con 0x7fd6dc006fd0 auth_method 0 2024-02-06T21:41:37.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7fd6e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7fd6e3fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7fd6e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f577b7fe640 1 -- 172.21.15.120:0/1468167761 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c031770 con 0x7f577c06f8e0 2024-02-06T21:41:37.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f577b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:37.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f577b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:37.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f577b7fe640 10 monclient: dump: 2024-02-06T21:41:37.980 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:37.980 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.980 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.980 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.980 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:37.981 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.981 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:37.981 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:37.981 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:37.981 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f577b7fe640 10 monclient: _finish_auth 0 2024-02-06T21:41:37.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f577b7fe640 10 monclient: _check_auth_tickets 2024-02-06T21:41:37.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f577b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:37.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f577b7fe640 1 -- 172.21.15.120:0/1468167761 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576c0318d0 con 0x7f577c06f8e0 2024-02-06T21:41:37.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f578304a640 5 monclient: authenticate success, global_id 4256 2024-02-06T21:41:37.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f578304a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:37.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f577b7fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:37.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f578304a640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:37.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f578304a640 4 set_mon_vals no callback set 2024-02-06T21:41:37.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f577b7fe640 1 -- 172.21.15.120:0/1468167761 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c031c60 con 0x7f577c06f8e0 2024-02-06T21:41:37.982 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f577b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:37.982 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f577b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:37.982 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.944+0000 7f577b7fe640 10 monclient: dump: 2024-02-06T21:41:37.982 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:37.982 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.982 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.982 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.982 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:37.982 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.982 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:37.982 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:37.982 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:37.982 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.945+0000 7f0389fe1640 1 --2- 172.21.15.120:0/3770364555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c007190 0x7f037c0c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:37.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.945+0000 7f578304a640 10 monclient: shutdown 2024-02-06T21:41:37.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.945+0000 7f578304a640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:37.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.945+0000 7f578304a640 1 -- 172.21.15.120:0/1468167761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f577c06f8e0 msgr2=0x7f577c06eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:37.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.945+0000 7f578304a640 1 --2- 172.21.15.120:0/1468167761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f577c06f8e0 0x7f577c06eb60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f576c0097d0 tx=0x7f576c02b810 comp rx=0 tx=0).stop 2024-02-06T21:41:37.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.945+0000 7f578304a640 1 -- 172.21.15.120:0/1468167761 shutdown_connections 2024-02-06T21:41:37.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.945+0000 7f578304a640 1 --2- 172.21.15.120:0/1468167761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f577c06f8e0 0x7f577c06eb60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.945+0000 7f578304a640 1 -- 172.21.15.120:0/1468167761 >> 172.21.15.120:0/1468167761 conn(0x7f577c0b07a0 msgr2=0x7f577c0b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:37.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.945+0000 7f578304a640 1 -- 172.21.15.120:0/1468167761 shutdown_connections 2024-02-06T21:41:37.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.945+0000 7f578304a640 1 -- 172.21.15.120:0/1468167761 wait complete. 2024-02-06T21:41:37.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.946+0000 7fd6e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:37.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.946+0000 7fd6e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:37.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.946+0000 7fd6e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:37.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.946+0000 7f578304a640 10 monclient: build_initial_monmap 2024-02-06T21:41:37.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.946+0000 7f578304a640 10 monclient: monmap: 2024-02-06T21:41:37.986 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:37.986 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:37.987 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:37.987 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:37.987 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:37.987 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.987 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:37.987 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:37.987 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:37.987 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.987 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.946+0000 7f578304a640 1 librados: starting msgr at 2024-02-06T21:41:37.987 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.946+0000 7f578304a640 1 librados: starting objecter 2024-02-06T21:41:37.987 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.946+0000 7f578304a640 1 Processor -- start 2024-02-06T21:41:37.987 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.946+0000 7f578304a640 1 -- start start 2024-02-06T21:41:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.946+0000 7f578304a640 1 librados: setting wanted keys 2024-02-06T21:41:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.946+0000 7f578304a640 1 librados: calling monclient init 2024-02-06T21:41:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.946+0000 7f578304a640 10 monclient: init 2024-02-06T21:41:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T21:41:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f577c06e780 0x7f577c140330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 10 monclient(hunting): picked mon.noname-b con 0x7f577c06e780 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f577c06f8e0 0x7f577c1470b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 10 monclient(hunting): picked mon.noname-a con 0x7f577c06f8e0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f577c0710d0 0x7f577c13a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 10 monclient(hunting): picked mon.noname-c con 0x7f577c0710d0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c146920 con 0x7f577c06f8e0 2024-02-06T21:41:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c1475f0 con 0x7f577c06e780 2024-02-06T21:41:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577c139b50 con 0x7f577c0710d0 2024-02-06T21:41:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f578304a640 10 monclient(hunting): authenticate will time out at 1328.284853s 2024-02-06T21:41:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f153bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f153bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7f153bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:37.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7fd6e3fff640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2024-02-06T21:41:37.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7fd6e3fff640 1 -- 172.21.15.120:0/3939976003 >> v2:172.21.15.120:3300/0 conn(0x7fd6dc033930 msgr2=0x7fd6dc0d24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7fd6e3fff640 1 --2- 172.21.15.120:0/3939976003 >> v2:172.21.15.120:3300/0 conn(0x7fd6dc033930 0x7fd6dc0d24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7fd6e3fff640 1 -- 172.21.15.120:0/3939976003 >> v1:172.21.15.120:6789/0 conn(0x7fd6dc0073b0 legacy=0x7fd6dc035f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7fd6e3fff640 10 monclient: _finish_hunting 0 2024-02-06T21:41:37.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7fd6e3fff640 1 monclient: found mon.noname-b 2024-02-06T21:41:37.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7fd6e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:37.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7fd6e3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:37.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.947+0000 7fd6e3fff640 1 -- 172.21.15.120:0/3939976003 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc035ac0 con 0x7fd6dc006fd0 2024-02-06T21:41:37.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.948+0000 7f0ba1017640 1 --2- 172.21.15.120:0/466123236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b9c06b590 0x7f0b9c13ec10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c071fa0 tx=0x7f0b9402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=686b33dc0a8844f2 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:37.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.949+0000 7f0389fe1640 10 monclient(hunting): get_auth_request con 0x7f037c007190 auth_method 0 2024-02-06T21:41:37.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.949+0000 7f0389fe1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:37.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.949+0000 7f0389fe1640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:37.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.949+0000 7f0389fe1640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:37.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.949+0000 7fd6e3fff640 1 --2- 172.21.15.120:0/3939976003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6dc006fd0 0x7fd6dc0cb150 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8009850 tx=0x7fd6d802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a50ec24543b02d71 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:37.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.949+0000 7f153bfff640 10 monclient(hunting): handle_auth_done global_id 4294 payload 274 2024-02-06T21:41:37.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.949+0000 7f153bfff640 1 -- 172.21.15.120:0/800412535 >> v1:172.21.15.135:6789/0 conn(0x7f153c0b5170 legacy=0x7f153c13f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:37.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.949+0000 7f153bfff640 1 -- 172.21.15.120:0/800412535 >> v1:172.21.15.120:6789/0 conn(0x7f153c13ff70 legacy=0x7f153c142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.949+0000 7f153bfff640 10 monclient: _finish_hunting 0 2024-02-06T21:41:37.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.949+0000 7f153bfff640 1 monclient: found mon.noname-c 2024-02-06T21:41:37.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.949+0000 7f153bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:37.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.949+0000 7f153bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T21:41:37.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.949+0000 7f153bfff640 1 -- 172.21.15.120:0/800412535 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c0ab570 con 0x7f153c0a97c0 2024-02-06T21:41:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.950+0000 7f0389fe1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.950+0000 7f0389fe1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.950+0000 7f0389fe1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 -- 172.21.15.120:0/3770364555 <== mon.1 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1207734078 0 0) 0x7f0370003520 con 0x7f037c007570 2024-02-06T21:41:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 -- 172.21.15.120:0/3770364555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c007190 msgr2=0x7f037c0c5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 --2- 172.21.15.120:0/3770364555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c007190 0x7f037c0c5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 -- 172.21.15.120:0/3770364555 >> v2:172.21.15.120:3300/0 conn(0x7f037c009030 msgr2=0x7f037c0d2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 --2- 172.21.15.120:0/3770364555 >> v2:172.21.15.120:3300/0 conn(0x7f037c009030 0x7f037c0d2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient: _finish_hunting 0 2024-02-06T21:41:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 monclient: found mon.noname-c-legacy 2024-02-06T21:41:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.135:6789/0 2024-02-06T21:41:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 -- 172.21.15.120:0/3770364555 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c0309e0 con 0x7f037c007570 2024-02-06T21:41:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient: _finish_auth 0 2024-02-06T21:41:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient: _check_auth_tickets 2024-02-06T21:41:37.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:37.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 -- 172.21.15.120:0/3770364555 <== mon.1 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2420367567 0 0) 0x7f0374001670 con 0x7f037c007570 2024-02-06T21:41:37.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:37.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-02-06T21:41:37.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient: dump: 2024-02-06T21:41:37.996 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:37.996 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:37.996 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.996 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:37.996 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:37.996 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:37.996 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:37.997 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:37.997 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:37.997 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:37.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] but i'm connected to v1:172.21.15.135:6789/0, reconnecting 2024-02-06T21:41:37.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:37.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 -- 172.21.15.120:0/3770364555 >> v1:172.21.15.135:6789/0 conn(0x7f037c007570 legacy=0x7f037c0c81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:37.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T21:41:37.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 --2- 172.21.15.120:0/3770364555 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0370003970 0x7f0370003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient(hunting): picked mon.b con 0x7f0370003970 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:37.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 --2- 172.21.15.120:0/3770364555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c007190 0x7f03700047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient(hunting): picked mon.c con 0x7f037c007190 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:37.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 --2- 172.21.15.120:0/3770364555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0370004d00 0x7f0370005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:37.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient(hunting): picked mon.a con 0x7f0370004d00 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:37.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 -- 172.21.15.120:0/3770364555 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c0c7cc0 con 0x7f0370004d00 2024-02-06T21:41:37.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 -- 172.21.15.120:0/3770364555 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c030700 con 0x7f0370003970 2024-02-06T21:41:37.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:37.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 1 -- 172.21.15.120:0/3770364555 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f037000bb60 con 0x7f037c007190 2024-02-06T21:41:37.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:37.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f0388fdf640 10 monclient(hunting): _finish_auth 0 2024-02-06T21:41:37.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.951+0000 7f57a2cda640 1 --2- 172.21.15.120:0/3694630743 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57940cf6e0 0x7f57940cfb10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f579c0691c0 tx=0x7f579c0a6710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fb16e7ed4d9dd3 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:37.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.952+0000 7f288259c640 1 --2- 172.21.15.120:0/3752993568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f287c02e1f0 0x7f287c02e5d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2878009870 tx=0x7f287802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7722f74e54aa6aa4 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:37.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.953+0000 7f153bfff640 1 --2- 172.21.15.120:0/800412535 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f153c0a97c0 0x7f153c142d20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1534009d90 tx=0x7f153402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dc37f232493518a server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:37.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.954+0000 7f5780de6640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f577c06e780 0x7f577c140330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fdf2a200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf2413ba80 0x7fdf241d3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e2ffd640 1 -- 172.21.15.120:0/3939976003 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d802e070 con 0x7fd6dc006fd0 2024-02-06T21:41:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e2ffd640 10 monclient: dump: 2024-02-06T21:41:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.002 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e2ffd640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e2ffd640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e2ffd640 1 -- 172.21.15.120:0/3939976003 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d8004030 con 0x7fd6dc006fd0 2024-02-06T21:41:38.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e91bc640 5 monclient: authenticate success, global_id 4268 2024-02-06T21:41:38.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e91bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:38.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e91bc640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:38.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e91bc640 4 set_mon_vals no callback set 2024-02-06T21:41:38.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e2ffd640 1 -- 172.21.15.120:0/3939976003 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8004420 con 0x7fd6dc006fd0 2024-02-06T21:41:38.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:38.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fdf2a200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf2413ba80 0x7fdf241d3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:56838/0 (socket says 172.21.15.120:56838) 2024-02-06T21:41:38.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fdf2a200640 1 -- 172.21.15.120:0/403538856 learned_addr learned my addr 172.21.15.120:0/403538856 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fdf2a200640 10 monclient(hunting): get_auth_request con 0x7fdf2413ba80 auth_method 0 2024-02-06T21:41:38.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fdf2a200640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fdf2a200640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fdf2a200640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.957+0000 7fd6e2ffd640 10 monclient: dump: 2024-02-06T21:41:38.004 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.004 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.005 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.005 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.005 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.005 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.005 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.005 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.005 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.005 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.005 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7f03897e0640 1 --2- 172.21.15.120:0/3770364555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c007190 0x7f03700047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fd6e91bc640 10 monclient: shutdown 2024-02-06T21:41:38.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fd6e91bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fd6e91bc640 1 -- 172.21.15.120:0/3939976003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6dc006fd0 msgr2=0x7fd6dc0cb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fd6e91bc640 1 --2- 172.21.15.120:0/3939976003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6dc006fd0 0x7fd6dc0cb150 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8009850 tx=0x7fd6d802b840 comp rx=0 tx=0).stop 2024-02-06T21:41:38.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fd6e91bc640 1 -- 172.21.15.120:0/3939976003 shutdown_connections 2024-02-06T21:41:38.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fd6e91bc640 1 --2- 172.21.15.120:0/3939976003 >> v2:172.21.15.120:3300/0 conn(0x7fd6dc033930 0x7fd6dc0d24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fd6e91bc640 1 --2- 172.21.15.120:0/3939976003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6dc006fd0 0x7fd6dc0cb150 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fd6e91bc640 1 -- 172.21.15.120:0/3939976003 >> 172.21.15.120:0/3939976003 conn(0x7fd6dc0061a0 msgr2=0x7fd6dc0065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fd6e91bc640 1 -- 172.21.15.120:0/3939976003 shutdown_connections 2024-02-06T21:41:38.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.958+0000 7fd6e91bc640 1 -- 172.21.15.120:0/3939976003 wait complete. 2024-02-06T21:41:38.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.959+0000 7f03897e0640 10 monclient(hunting): get_auth_request con 0x7f037c007190 auth_method 0 2024-02-06T21:41:38.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.959+0000 7f03897e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.959+0000 7f03897e0640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.959+0000 7f03897e0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-02-06T21:41:38.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.959+0000 7f5780de6640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f577c06e780 0x7f577c140330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38548/0 (socket says 172.21.15.120:38548) 2024-02-06T21:41:38.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.959+0000 7f5780de6640 1 -- 172.21.15.120:0/3775077712 learned_addr learned my addr 172.21.15.120:0/3775077712 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.959+0000 7f5780de6640 10 monclient(hunting): get_auth_request con 0x7f577c06e780 auth_method 0 2024-02-06T21:41:38.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.959+0000 7f5780de6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.959+0000 7f5780de6640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.959+0000 7f5780de6640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.959+0000 7fd6e91bc640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.959+0000 7fd6e91bc640 10 monclient: monmap: 2024-02-06T21:41:38.008 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.009 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:38.009 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:38.009 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:38.009 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.009 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.009 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:38.009 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:38.009 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:38.009 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.009 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.959+0000 7f57815e7640 1 --2- 172.21.15.120:0/3775077712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f577c0710d0 0x7f577c13a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.960+0000 7fd6e91bc640 1 librados: starting msgr at 2024-02-06T21:41:38.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.960+0000 7fd6e91bc640 1 librados: starting objecter 2024-02-06T21:41:38.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.960+0000 7fd6e91bc640 1 Processor -- start 2024-02-06T21:41:38.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.960+0000 7fd6e91bc640 1 -- start start 2024-02-06T21:41:38.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.960+0000 7fd6e91bc640 1 librados: setting wanted keys 2024-02-06T21:41:38.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.960+0000 7fdf2a200640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.960+0000 7fdf2a200640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.960+0000 7fdf2a200640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.960+0000 7fd6e91bc640 1 librados: calling monclient init 2024-02-06T21:41:38.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.960+0000 7fd6e91bc640 10 monclient: init 2024-02-06T21:41:38.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.960+0000 7fdf299ff640 1 --2- 172.21.15.120:0/403538856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf2406f440 0x7fdf240acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T21:41:38.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6dc006fd0 0x7fd6dc1605f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 10 monclient(hunting): picked mon.noname-b con 0x7fd6dc006fd0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:38.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6dc033930 0x7fd6dc02de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 10 monclient(hunting): picked mon.noname-c con 0x7fd6dc033930 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6dc02eae0 0x7fd6dc02ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 10 monclient(hunting): picked mon.noname-a con 0x7fd6dc02eae0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:38.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc030610 con 0x7fd6dc02eae0 2024-02-06T21:41:38.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc02e3a0 con 0x7fd6dc006fd0 2024-02-06T21:41:38.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc02e640 con 0x7fd6dc033930 2024-02-06T21:41:38.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7fd6e91bc640 10 monclient(hunting): authenticate will time out at 1328.298780s 2024-02-06T21:41:38.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7f03897e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7f5780de6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7f5780de6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7f5780de6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7f03897e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.961+0000 7f03897e0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-02-06T21:41:38.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.962+0000 7f57815e7640 10 monclient(hunting): get_auth_request con 0x7f577c0710d0 auth_method 0 2024-02-06T21:41:38.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.962+0000 7f57815e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.962+0000 7f57815e7640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.962+0000 7f57815e7640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.962+0000 7fdf299ff640 10 monclient(hunting): get_auth_request con 0x7fdf2406f440 auth_method 0 2024-02-06T21:41:38.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.962+0000 7fdf299ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.962+0000 7fdf299ff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.962+0000 7fdf299ff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288159a640 1 -- 172.21.15.120:0/3752993568 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2878030840 con 0x7f287c02e1f0 2024-02-06T21:41:38.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:38.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288159a640 10 monclient: dump: 2024-02-06T21:41:38.018 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.018 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.018 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.018 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.018 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.018 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.018 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.018 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.019 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.019 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288159a640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288159a640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288359e640 5 monclient: authenticate success, global_id 4293 2024-02-06T21:41:38.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:38.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288159a640 1 -- 172.21.15.120:0/3752993568 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28780309a0 con 0x7f287c02e1f0 2024-02-06T21:41:38.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288159a640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288359e640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:38.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288359e640 4 set_mon_vals no callback set 2024-02-06T21:41:38.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288159a640 1 -- 172.21.15.120:0/3752993568 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2878030d30 con 0x7f287c02e1f0 2024-02-06T21:41:38.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:38.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.963+0000 7f288159a640 10 monclient: dump: 2024-02-06T21:41:38.020 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.020 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.020 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.020 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.020 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.020 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.020 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.020 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.020 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.020 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7fdf291fe640 1 --2- 172.21.15.120:0/403538856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf24072010 0x7fdf241d3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7f288359e640 10 monclient: shutdown 2024-02-06T21:41:38.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7f0389fe1640 1 --2- 172.21.15.120:0/3770364555 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0370003970 0x7f0370003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7f288359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7f288359e640 1 -- 172.21.15.120:0/3752993568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f287c02e1f0 msgr2=0x7f287c02e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7f288359e640 1 --2- 172.21.15.120:0/3752993568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f287c02e1f0 0x7f287c02e5d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2878009870 tx=0x7f287802b980 comp rx=0 tx=0).stop 2024-02-06T21:41:38.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7fdf2a200640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2024-02-06T21:41:38.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7fdf2a200640 1 -- 172.21.15.120:0/403538856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf2406f440 msgr2=0x7fdf240acab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7f288359e640 1 -- 172.21.15.120:0/3752993568 shutdown_connections 2024-02-06T21:41:38.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7f288359e640 1 --2- 172.21.15.120:0/3752993568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f287c0cfc80 0x7f287c0d6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7f288359e640 1 --2- 172.21.15.120:0/3752993568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f287c02e1f0 0x7f287c02e5d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7f288359e640 1 -- 172.21.15.120:0/3752993568 >> 172.21.15.120:0/3752993568 conn(0x7f287c036610 msgr2=0x7f287c005e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7fdf2a200640 1 --2- 172.21.15.120:0/403538856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf2406f440 0x7fdf240acab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7fdf2a200640 1 -- 172.21.15.120:0/403538856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf24072010 msgr2=0x7fdf241d3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7fdf2a200640 1 --2- 172.21.15.120:0/403538856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf24072010 0x7fdf241d3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7fdf2a200640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7fdf2a200640 1 monclient: found mon.noname-c 2024-02-06T21:41:38.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7fdf2a200640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7fdf2a200640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7fdf2a200640 1 -- 172.21.15.120:0/403538856 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c00d050 con 0x7fdf2413ba80 2024-02-06T21:41:38.024 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.964+0000 7f288359e640 1 -- 172.21.15.120:0/3752993568 shutdown_connections 2024-02-06T21:41:38.024 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.965+0000 7f288359e640 1 -- 172.21.15.120:0/3752993568 wait complete. 2024-02-06T21:41:38.024 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0389fe1640 10 monclient(hunting): get_auth_request con 0x7f0370003970 auth_method 0 2024-02-06T21:41:38.024 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0389fe1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0389fe1640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0389fe1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-02-06T21:41:38.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f288359e640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f288359e640 10 monclient: monmap: 2024-02-06T21:41:38.028 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.028 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:38.028 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:38.028 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:38.028 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.028 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.028 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:38.028 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:38.028 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:38.028 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f288359e640 1 librados: starting msgr at 2024-02-06T21:41:38.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f288359e640 1 librados: starting objecter 2024-02-06T21:41:38.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f288359e640 1 Processor -- start 2024-02-06T21:41:38.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0b93fff640 1 -- 172.21.15.120:0/466123236 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9402ca80 con 0x7f0b9c06b590 2024-02-06T21:41:38.030 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0b93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.030 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0b93fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:38.030 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0b93fff640 10 monclient: dump: 2024-02-06T21:41:38.030 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.030 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.030 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.030 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.030 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.030 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.030 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.030 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.030 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.030 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0b93fff640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0b93fff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0b93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0ba327b640 5 monclient: authenticate success, global_id 4287 2024-02-06T21:41:38.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0ba327b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:38.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0b93fff640 1 -- 172.21.15.120:0/466123236 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9402cbe0 con 0x7f0b9c06b590 2024-02-06T21:41:38.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0b93fff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0ba327b640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:38.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0ba327b640 4 set_mon_vals no callback set 2024-02-06T21:41:38.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0b93fff640 1 -- 172.21.15.120:0/466123236 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b940343f0 con 0x7f0b9c06b590 2024-02-06T21:41:38.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0b93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0b93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:38.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f0b93fff640 10 monclient: dump: 2024-02-06T21:41:38.031 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.032 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.032 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.032 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.032 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.032 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.032 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.032 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.032 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.032 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f577bfff640 1 --2- 172.21.15.120:0/3775077712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f577c06f8e0 0x7f577c1470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f0ba327b640 10 monclient: shutdown 2024-02-06T21:41:38.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f0ba327b640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f0ba327b640 1 -- 172.21.15.120:0/466123236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b9c06b590 msgr2=0x7f0b9c13ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f0ba327b640 1 --2- 172.21.15.120:0/466123236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b9c06b590 0x7f0b9c13ec10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c071fa0 tx=0x7f0b9402bee0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f0ba327b640 1 -- 172.21.15.120:0/466123236 shutdown_connections 2024-02-06T21:41:38.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f0ba327b640 1 --2- 172.21.15.120:0/466123236 >> v2:172.21.15.135:3300/0 conn(0x7f0b9c06ca30 0x7f0b9c13f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f0ba327b640 1 --2- 172.21.15.120:0/466123236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b9c06b590 0x7f0b9c13ec10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f0ba327b640 1 -- 172.21.15.120:0/466123236 >> 172.21.15.120:0/466123236 conn(0x7f0b9c0b0ae0 msgr2=0x7f0b9c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f288359e640 1 -- start start 2024-02-06T21:41:38.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f288359e640 1 librados: setting wanted keys 2024-02-06T21:41:38.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f288359e640 1 librados: calling monclient init 2024-02-06T21:41:38.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.966+0000 7f288359e640 10 monclient: init 2024-02-06T21:41:38.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f0ba327b640 1 -- 172.21.15.120:0/466123236 shutdown_connections 2024-02-06T21:41:38.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f0ba327b640 1 -- 172.21.15.120:0/466123236 wait complete. 2024-02-06T21:41:38.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T21:41:38.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f287c02e1f0 0x7f287c0caf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 10 monclient(hunting): picked mon.noname-b con 0x7f287c02e1f0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:38.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f287c032960 0x7f287c0cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 10 monclient(hunting): picked mon.noname-a con 0x7f287c032960 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:38.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f287c0cfc80 0x7f287c0ce070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 10 monclient(hunting): picked mon.noname-c con 0x7f287c0cfc80 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c0c05b0 con 0x7f287c032960 2024-02-06T21:41:38.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c0cd050 con 0x7f287c02e1f0 2024-02-06T21:41:38.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287c0cd330 con 0x7f287c0cfc80 2024-02-06T21:41:38.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.967+0000 7f288359e640 10 monclient(hunting): authenticate will time out at 1328.305195s 2024-02-06T21:41:38.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f0ba327b640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f0ba327b640 10 monclient: monmap: 2024-02-06T21:41:38.039 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.039 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:38.040 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:38.040 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:38.040 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.040 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.040 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:38.040 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:38.040 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:38.040 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f0ba327b640 1 librados: starting msgr at 2024-02-06T21:41:38.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f0ba327b640 1 librados: starting objecter 2024-02-06T21:41:38.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f0ba327b640 1 Processor -- start 2024-02-06T21:41:38.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f0ba327b640 1 -- start start 2024-02-06T21:41:38.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f0ba327b640 1 librados: setting wanted keys 2024-02-06T21:41:38.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f0ba327b640 1 librados: calling monclient init 2024-02-06T21:41:38.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f0ba327b640 10 monclient: init 2024-02-06T21:41:38.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f577bfff640 10 monclient(hunting): get_auth_request con 0x7f577c06f8e0 auth_method 0 2024-02-06T21:41:38.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f577bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f577bfff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f577bfff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f5780de6640 10 monclient(hunting): handle_auth_done global_id 4297 payload 824 2024-02-06T21:41:38.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f5780de6640 1 -- 172.21.15.120:0/3775077712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f577c0710d0 msgr2=0x7f577c13a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f5780de6640 1 --2- 172.21.15.120:0/3775077712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f577c0710d0 0x7f577c13a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f5780de6640 1 -- 172.21.15.120:0/3775077712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f577c06f8e0 msgr2=0x7f577c1470b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f5780de6640 1 --2- 172.21.15.120:0/3775077712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f577c06f8e0 0x7f577c1470b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f5780de6640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f5780de6640 1 monclient: found mon.noname-b 2024-02-06T21:41:38.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f5780de6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f5780de6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.968+0000 7f5780de6640 1 -- 172.21.15.120:0/3775077712 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c009400 con 0x7f577c06e780 2024-02-06T21:41:38.043 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.043 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 10 monclient: _add_conns ranks=[0,2,1] 2024-02-06T21:41:38.043 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b9c06b590 0x7f0b9c06deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.043 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 10 monclient(hunting): picked mon.noname-a con 0x7f0b9c06b590 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:38.043 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b9c06ca30 0x7f0b9c0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.043 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 10 monclient(hunting): picked mon.noname-c con 0x7f0b9c06ca30 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.043 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b9c0a9ce0 0x7f0b9c0717f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.043 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 10 monclient(hunting): picked mon.noname-b con 0x7f0b9c0a9ce0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:38.043 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.044 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c06f340 con 0x7f0b9c06b590 2024-02-06T21:41:38.044 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.044 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c06f5b0 con 0x7f0b9c0a9ce0 2024-02-06T21:41:38.044 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.044 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c06f890 con 0x7f0b9c06ca30 2024-02-06T21:41:38.044 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.044 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.969+0000 7f0ba327b640 10 monclient(hunting): authenticate will time out at 1328.306682s 2024-02-06T21:41:38.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7f03897e0640 10 monclient(hunting): handle_auth_done global_id 4285 payload 293 2024-02-06T21:41:38.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7f03897e0640 1 -- 172.21.15.120:0/3770364555 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0370003970 msgr2=0x7f0370003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:38.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7f03897e0640 1 --2- 172.21.15.120:0/3770364555 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0370003970 0x7f0370003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7f03897e0640 1 -- 172.21.15.120:0/3770364555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0370004d00 msgr2=0x7f0370005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:38.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7f03897e0640 1 --2- 172.21.15.120:0/3770364555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0370004d00 0x7f0370005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7f03897e0640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7f03897e0640 1 monclient: found mon.c 2024-02-06T21:41:38.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7f03897e0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7f03897e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7f03897e0640 1 -- 172.21.15.120:0/3770364555 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0370003520 con 0x7f037c007190 2024-02-06T21:41:38.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7f03897e0640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7f03897e0640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7f03897e0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7f038c245640 5 monclient: authenticate success, global_id 4285 2024-02-06T21:41:38.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7f038c245640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:38.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.970+0000 7fd6e89bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6dc02eae0 0x7fd6dc02ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.972+0000 7f5780de6640 1 --2- 172.21.15.120:0/3775077712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f577c06e780 0x7f577c140330 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f577000c430 tx=0x7f577000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.047 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.972+0000 7f57797fa640 10 client.?.objecter ms_handle_connect 0x7f577c06e780 2024-02-06T21:41:38.047 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.972+0000 7f57797fa640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:38.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.972+0000 7fd6e89bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6dc02eae0 0x7fd6dc02ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60152/0 (socket says 172.21.15.120:60152) 2024-02-06T21:41:38.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.972+0000 7fd6e89bb640 1 -- 172.21.15.120:0/2285512625 learned_addr learned my addr 172.21.15.120:0/2285512625 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.972+0000 7fd6e89bb640 10 monclient(hunting): get_auth_request con 0x7fd6dc02eae0 auth_method 0 2024-02-06T21:41:38.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.972+0000 7fd6e89bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.972+0000 7fd6e89bb640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.972+0000 7fd6e89bb640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.974+0000 7fd6e89bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.974+0000 7fd6e89bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.974+0000 7fd6e89bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.975+0000 7f03897e0640 1 --2- 172.21.15.120:0/3770364555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c007190 0x7f03700047c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f037400c160 tx=0x7f03700057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfadb49a4ba415f7 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.975+0000 7fdf2a200640 1 --2- 172.21.15.120:0/403538856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf2413ba80 0x7fdf241d3d70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c03a040 tx=0x7fdf1c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.975+0000 7fdf1a7fc640 10 client.?.objecter ms_handle_connect 0x7fdf2413ba80 2024-02-06T21:41:38.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.975+0000 7fdf1a7fc640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:38.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f153a7fc640 1 -- 172.21.15.120:0/800412535 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153402cc60 con 0x7f153c0a97c0 2024-02-06T21:41:38.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f153a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f153a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:38.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f153a7fc640 10 monclient: dump: 2024-02-06T21:41:38.050 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.050 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.050 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.050 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.050 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.050 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.050 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.050 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.050 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f153a7fc640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f153a7fc640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f153a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f153a7fc640 1 -- 172.21.15.120:0/800412535 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153402cdc0 con 0x7f153c0a97c0 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f153a7fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f153a7fc640 1 -- 172.21.15.120:0/800412535 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1534034660 con 0x7f153c0a97c0 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f153a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f153a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f153a7fc640 10 monclient: dump: 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.051 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.052 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.052 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.052 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.052 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.052 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f1540899640 5 monclient: authenticate success, global_id 4294 2024-02-06T21:41:38.052 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f1540899640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:38.052 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7fd6e3fff640 1 --2- 172.21.15.120:0/2285512625 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6dc006fd0 0x7fd6dc1605f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.053 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.976+0000 7f1540899640 4 set_mon_vals no callback set 2024-02-06T21:41:38.053 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7f1540899640 10 monclient: shutdown 2024-02-06T21:41:38.053 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7f1540899640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.053 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7f1540899640 1 -- 172.21.15.120:0/800412535 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f153c0a97c0 msgr2=0x7f153c142d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.053 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7f1540899640 1 --2- 172.21.15.120:0/800412535 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f153c0a97c0 0x7f153c142d20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1534009d90 tx=0x7f153402beb0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.053 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7f1540899640 1 -- 172.21.15.120:0/800412535 shutdown_connections 2024-02-06T21:41:38.054 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7f1540899640 1 --2- 172.21.15.120:0/800412535 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f153c0a97c0 0x7f153c142d20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.054 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7f1540899640 1 -- 172.21.15.120:0/800412535 >> 172.21.15.120:0/800412535 conn(0x7f153c066df0 msgr2=0x7f153c067200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.054 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7f1540899640 1 -- 172.21.15.120:0/800412535 shutdown_connections 2024-02-06T21:41:38.054 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7f1540899640 1 -- 172.21.15.120:0/800412535 wait complete. 2024-02-06T21:41:38.054 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7fd6e3fff640 10 monclient(hunting): get_auth_request con 0x7fd6dc006fd0 auth_method 0 2024-02-06T21:41:38.054 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7fd6e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.054 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7fd6e3fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.055 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7fd6e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.055 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7fd6e89bb640 10 monclient(hunting): handle_auth_done global_id 4299 payload 824 2024-02-06T21:41:38.055 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7fd6e89bb640 1 -- 172.21.15.120:0/2285512625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6dc033930 msgr2=0x7fd6dc02de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.055 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7fd6e89bb640 1 --2- 172.21.15.120:0/2285512625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6dc033930 0x7fd6dc02de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.055 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7fd6e89bb640 1 -- 172.21.15.120:0/2285512625 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6dc006fd0 msgr2=0x7fd6dc1605f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.055 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7fd6e89bb640 1 --2- 172.21.15.120:0/2285512625 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6dc006fd0 0x7fd6dc1605f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.055 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7fd6e89bb640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.055 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7fd6e89bb640 1 monclient: found mon.noname-a 2024-02-06T21:41:38.055 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7fd6e89bb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.055 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7fd6e89bb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.055 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.977+0000 7fd6e89bb640 1 -- 172.21.15.120:0/2285512625 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d8009460 con 0x7fd6dc02eae0 2024-02-06T21:41:38.056 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f1540899640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.056 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f1540899640 10 monclient: monmap: 2024-02-06T21:41:38.056 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.056 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:38.056 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:38.057 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:38.057 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.057 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.057 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:38.057 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:38.057 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:38.057 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f1540899640 1 librados: starting msgr at 2024-02-06T21:41:38.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f1540899640 1 librados: starting objecter 2024-02-06T21:41:38.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f1540899640 1 Processor -- start 2024-02-06T21:41:38.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f1540899640 1 -- start start 2024-02-06T21:41:38.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f1540899640 1 librados: setting wanted keys 2024-02-06T21:41:38.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f1540899640 1 librados: calling monclient init 2024-02-06T21:41:38.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f1540899640 10 monclient: init 2024-02-06T21:41:38.058 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a14d7640 1 -- 172.21.15.120:0/3694630743 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c0ab7d0 con 0x7f57940cf6e0 2024-02-06T21:41:38.058 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a14d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.058 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a14d7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:38.058 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a14d7640 10 monclient: dump: 2024-02-06T21:41:38.059 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.059 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.059 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.059 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.059 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.059 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.059 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.059 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.059 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.059 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a14d7640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a14d7640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a14d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a14d7640 1 -- 172.21.15.120:0/3694630743 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579c0a9050 con 0x7f57940cf6e0 2024-02-06T21:41:38.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a473d640 5 monclient: authenticate success, global_id 4288 2024-02-06T21:41:38.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a473d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:38.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a14d7640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a473d640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:38.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a14d7640 1 -- 172.21.15.120:0/3694630743 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c0abc30 con 0x7f57940cf6e0 2024-02-06T21:41:38.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a473d640 4 set_mon_vals no callback set 2024-02-06T21:41:38.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a14d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a14d7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:38.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.978+0000 7f57a14d7640 10 monclient: dump: 2024-02-06T21:41:38.060 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.060 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.060 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.060 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.061 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.061 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.061 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.061 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.061 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.061 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 10 monclient: _add_conns ranks=[0,1,2] 2024-02-06T21:41:38.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f153c0a97c0 0x7f153c0b7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 10 monclient(hunting): picked mon.noname-a con 0x7f153c0a97c0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:38.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f153c0b5170 0x7f153c138b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 10 monclient(hunting): picked mon.noname-b con 0x7f153c0b5170 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:38.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f153c13ff70 0x7f153c139bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 10 monclient(hunting): picked mon.noname-c con 0x7f153c13ff70 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c13a970 con 0x7f153c0a97c0 2024-02-06T21:41:38.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c139710 con 0x7f153c0b5170 2024-02-06T21:41:38.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c13abe0 con 0x7f153c13ff70 2024-02-06T21:41:38.063 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.063 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f57a473d640 10 monclient: shutdown 2024-02-06T21:41:38.063 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f57a473d640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.063 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f57a473d640 1 -- 172.21.15.120:0/3694630743 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57940cf6e0 msgr2=0x7f57940cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.063 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f57a473d640 1 --2- 172.21.15.120:0/3694630743 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57940cf6e0 0x7f57940cfb10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f579c0691c0 tx=0x7f579c0a6710 comp rx=0 tx=0).stop 2024-02-06T21:41:38.063 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f1540899640 10 monclient(hunting): authenticate will time out at 1328.317016s 2024-02-06T21:41:38.065 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f57a473d640 1 -- 172.21.15.120:0/3694630743 shutdown_connections 2024-02-06T21:41:38.065 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f57a473d640 1 --2- 172.21.15.120:0/3694630743 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57940cf6e0 0x7f57940cfb10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.065 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f57a473d640 1 --2- 172.21.15.120:0/3694630743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5794007440 0x7f579400a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.065 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f57a473d640 1 -- 172.21.15.120:0/3694630743 >> 172.21.15.120:0/3694630743 conn(0x7f57940061f0 msgr2=0x7f5794006600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.065 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f57a473d640 1 -- 172.21.15.120:0/3694630743 shutdown_connections 2024-02-06T21:41:38.066 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.979+0000 7f57a473d640 1 -- 172.21.15.120:0/3694630743 wait complete. 2024-02-06T21:41:38.066 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.980+0000 7f57a473d640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.066 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.980+0000 7f57a473d640 10 monclient: monmap: 2024-02-06T21:41:38.066 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.066 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:38.066 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:38.066 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:38.066 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.066 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.066 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:38.066 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:38.066 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:38.066 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.066 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.981+0000 7f57a473d640 1 librados: starting msgr at 2024-02-06T21:41:38.066 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.981+0000 7f57a473d640 1 librados: starting objecter 2024-02-06T21:41:38.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.981+0000 7f57a473d640 1 Processor -- start 2024-02-06T21:41:38.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.981+0000 7f57a473d640 1 -- start start 2024-02-06T21:41:38.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.981+0000 7f57a473d640 1 librados: setting wanted keys 2024-02-06T21:41:38.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.981+0000 7f57a473d640 1 librados: calling monclient init 2024-02-06T21:41:38.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.981+0000 7f57a473d640 10 monclient: init 2024-02-06T21:41:38.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.981+0000 7f57a473d640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.981+0000 7f57a473d640 10 monclient: _add_conns ranks=[0,2,1] 2024-02-06T21:41:38.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.981+0000 7f57a473d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5794007440 0x7f57940ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.981+0000 7f57a473d640 10 monclient(hunting): picked mon.noname-a con 0x7f5794007440 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:38.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.981+0000 7f57a473d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57940c8f60 0x7f57940c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.982+0000 7f57a473d640 10 monclient(hunting): picked mon.noname-c con 0x7f57940c8f60 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.982+0000 7f57a473d640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57940cf6e0 0x7f57940cb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.982+0000 7f57a473d640 10 monclient(hunting): picked mon.noname-b con 0x7f57940cf6e0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:38.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.982+0000 7f57a473d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.068 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.982+0000 7f57a473d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57940d11f0 con 0x7f5794007440 2024-02-06T21:41:38.068 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.982+0000 7f57a473d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.068 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.982+0000 7f57a473d640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57940ca800 con 0x7f57940cf6e0 2024-02-06T21:41:38.068 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.982+0000 7f57a473d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.068 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.982+0000 7f57a473d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57940ca3a0 con 0x7f57940c8f60 2024-02-06T21:41:38.068 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.982+0000 7f57a473d640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.068 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.982+0000 7f57a473d640 10 monclient(hunting): authenticate will time out at 1328.319634s 2024-02-06T21:41:38.068 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.983+0000 7fd6e89bb640 1 --2- 172.21.15.120:0/2285512625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6dc02eae0 0x7fd6dc02ef40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd6e404fd40 tx=0x7fd6e4069f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.068 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.983+0000 7fd6e0ff9640 10 client.?.objecter ms_handle_connect 0x7fd6dc02eae0 2024-02-06T21:41:38.068 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.983+0000 7fd6e0ff9640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:38.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.983+0000 7f0388fdf640 1 -- 172.21.15.120:0/3770364555 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0374005ad0 con 0x7f037c007190 2024-02-06T21:41:38.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.983+0000 7f0388fdf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.983+0000 7f0388fdf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:38.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.983+0000 7f0388fdf640 10 monclient: dump: 2024-02-06T21:41:38.069 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.069 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.069 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.070 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.070 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.070 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.070 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.070 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.070 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.070 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.070 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.983+0000 7f0388fdf640 1 -- 172.21.15.120:0/3770364555 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0374005c30 con 0x7f037c007190 2024-02-06T21:41:38.070 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.983+0000 7f0388fdf640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.070 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.983+0000 7f0388fdf640 1 -- 172.21.15.120:0/3770364555 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037402ca20 con 0x7f037c007190 2024-02-06T21:41:38.070 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.983+0000 7f0388fdf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.070 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.983+0000 7f0388fdf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:38.070 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.983+0000 7f0388fdf640 10 monclient: dump: 2024-02-06T21:41:38.070 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.071 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.071 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.071 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.071 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.071 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.071 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.071 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.071 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.071 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.984+0000 7f038c245640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:38.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.984+0000 7f038c245640 4 set_mon_vals no callback set 2024-02-06T21:41:38.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.984+0000 7f038c245640 10 monclient: shutdown 2024-02-06T21:41:38.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.984+0000 7f038c245640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.984+0000 7f038c245640 1 -- 172.21.15.120:0/3770364555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c007190 msgr2=0x7f03700047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.984+0000 7f038c245640 1 --2- 172.21.15.120:0/3770364555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c007190 0x7f03700047c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f037400c160 tx=0x7f03700057f0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.984+0000 7f0ba1017640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b9c06b590 0x7f0b9c06deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.984+0000 7f038a7e2640 1 -- 172.21.15.120:0/3770364555 reap_dead start 2024-02-06T21:41:38.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.984+0000 7f038c245640 1 -- 172.21.15.120:0/3770364555 shutdown_connections 2024-02-06T21:41:38.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.984+0000 7f038c245640 1 -- 172.21.15.120:0/3770364555 >> 172.21.15.120:0/3770364555 conn(0x7f037c006250 msgr2=0x7f037c006660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.984+0000 7f038c245640 1 -- 172.21.15.120:0/3770364555 shutdown_connections 2024-02-06T21:41:38.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.984+0000 7f038c245640 1 -- 172.21.15.120:0/3770364555 wait complete. 2024-02-06T21:41:38.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf1a7fc640 1 -- 172.21.15.120:0/403538856 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c046070 con 0x7fdf2413ba80 2024-02-06T21:41:38.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf1a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:38.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf1a7fc640 10 monclient: dump: 2024-02-06T21:41:38.074 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.074 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.074 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.074 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.074 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.074 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.075 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.075 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.075 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.075 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf1a7fc640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf1a7fc640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf1a7fc640 1 -- 172.21.15.120:0/403538856 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1c00be90 con 0x7fdf2413ba80 2024-02-06T21:41:38.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf1a7fc640 1 -- 172.21.15.120:0/403538856 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c0417e0 con 0x7fdf2413ba80 2024-02-06T21:41:38.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf2bc63640 5 monclient: authenticate success, global_id 4274 2024-02-06T21:41:38.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf2bc63640 10 monclient: _renew_subs 2024-02-06T21:41:38.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf2bc63640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf1bfff640 4 set_mon_vals no callback set 2024-02-06T21:41:38.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf2bc63640 1 -- 172.21.15.120:0/403538856 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf241d42b0 con 0x7fdf2413ba80 2024-02-06T21:41:38.076 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.076 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:38.076 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf1a7fc640 10 monclient: dump: 2024-02-06T21:41:38.076 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.076 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.076 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.076 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.076 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.076 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.076 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.076 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.076 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.076 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.076 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf2bc63640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.077 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf2bc63640 10 monclient: _renew_subs 2024-02-06T21:41:38.077 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf2bc63640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.077 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf2bc63640 1 -- 172.21.15.120:0/403538856 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf240adbb0 con 0x7fdf2413ba80 2024-02-06T21:41:38.077 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7fdf2bc63640 1 librados: init done 2024-02-06T21:41:38.077 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7f0ba0816640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b9c06ca30 0x7f0b9c0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.077 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7f038c245640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.077 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7f038c245640 10 monclient: monmap: 2024-02-06T21:41:38.078 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.078 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:38.078 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:38.078 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:38.078 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.078 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.078 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:38.078 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:38.078 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:38.078 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.078 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.985+0000 7f288259c640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f287c02e1f0 0x7f287c0caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.079 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 1 librados: starting msgr at 2024-02-06T21:41:38.079 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 1 librados: starting objecter 2024-02-06T21:41:38.079 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7fdf2bc63640 10 client.4274.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.079 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7fdf2bc63640 15 client.4274.objecter _session_command_op_assign -1 1 2024-02-06T21:41:38.079 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7fdf2bc63640 20 client.4274.objecter _get_session osd=-1 returning homeless 2024-02-06T21:41:38.080 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7fdf2bc63640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.080 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7fdf2bc63640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.080 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7fdf2bc63640 1 -- 172.21.15.120:0/403538856 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdf24072e60 con 0x7fdf2413ba80 2024-02-06T21:41:38.080 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 1 Processor -- start 2024-02-06T21:41:38.080 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f288259c640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f287c02e1f0 0x7f287c0caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38570/0 (socket says 172.21.15.120:38570) 2024-02-06T21:41:38.081 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 1 -- start start 2024-02-06T21:41:38.081 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 1 librados: setting wanted keys 2024-02-06T21:41:38.081 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f288259c640 1 -- 172.21.15.120:0/214071414 learned_addr learned my addr 172.21.15.120:0/214071414 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.081 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f288259c640 10 monclient(hunting): get_auth_request con 0x7f287c02e1f0 auth_method 0 2024-02-06T21:41:38.081 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f288259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.082 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 1 librados: calling monclient init 2024-02-06T21:41:38.082 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 10 monclient: init 2024-02-06T21:41:38.082 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f288259c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.082 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f288259c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f0ba1818640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b9c0a9ce0 0x7f0b9c0717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f0ba1017640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b9c06b590 0x7f0b9c06deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60164/0 (socket says 172.21.15.120:60164) 2024-02-06T21:41:38.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f0ba1017640 1 -- 172.21.15.120:0/734218230 learned_addr learned my addr 172.21.15.120:0/734218230 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f0ba1017640 10 monclient(hunting): get_auth_request con 0x7f0b9c06b590 auth_method 0 2024-02-06T21:41:38.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f0ba1017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f0ba1017640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f0ba1017640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.084 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.084 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 10 monclient: _add_conns ranks=[2,0,1] 2024-02-06T21:41:38.084 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f153bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f153c13ff70 0x7f153c139bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.084 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c007190 0x7f037c034c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 10 monclient(hunting): picked mon.noname-c con 0x7f037c007190 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f037c032110 0x7f037c0a3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 10 monclient(hunting): picked mon.noname-a con 0x7f037c032110 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:38.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f037c0a3860 0x7f037c15d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 10 monclient(hunting): picked mon.noname-b con 0x7f037c0a3860 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:38.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c0a4cb0 con 0x7f037c032110 2024-02-06T21:41:38.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c15d020 con 0x7f037c0a3860 2024-02-06T21:41:38.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f037c15da80 con 0x7f037c007190 2024-02-06T21:41:38.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.986+0000 7f038c245640 10 monclient(hunting): authenticate will time out at 1328.324498s 2024-02-06T21:41:38.086 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f288259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.086 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f288259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.086 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f288259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.086 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f153bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f153c13ff70 0x7f153c139bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:56910/0 (socket says 172.21.15.120:56910) 2024-02-06T21:41:38.086 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f153bfff640 1 -- 172.21.15.120:0/3429897340 learned_addr learned my addr 172.21.15.120:0/3429897340 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f153bfff640 10 monclient(hunting): get_auth_request con 0x7f153c13ff70 auth_method 0 2024-02-06T21:41:38.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f153bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f153bfff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f153bfff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f0ba1818640 10 monclient(hunting): get_auth_request con 0x7f0b9c0a9ce0 auth_method 0 2024-02-06T21:41:38.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f0ba1818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f0ba1818640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f0ba1818640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f0ba0816640 10 monclient(hunting): get_auth_request con 0x7f0b9c06ca30 auth_method 0 2024-02-06T21:41:38.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f0ba0816640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f0ba0816640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.987+0000 7f0ba0816640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.988+0000 7f2881d9b640 1 --2- 172.21.15.120:0/214071414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f287c032960 0x7f287c0cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.989+0000 7f0ba1017640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.989+0000 7f0ba1017640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.989+0000 7f0ba1017640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.990+0000 7f2881d9b640 10 monclient(hunting): get_auth_request con 0x7f287c032960 auth_method 0 2024-02-06T21:41:38.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.990+0000 7f2881d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.990+0000 7f2881d9b640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.990+0000 7f2881d9b640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.990+0000 7f288259c640 10 monclient(hunting): handle_auth_done global_id 4306 payload 824 2024-02-06T21:41:38.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.990+0000 7f288259c640 1 -- 172.21.15.120:0/214071414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f287c0cfc80 msgr2=0x7f287c0ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.990+0000 7f288259c640 1 --2- 172.21.15.120:0/214071414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f287c0cfc80 0x7f287c0ce070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.990+0000 7f288259c640 1 -- 172.21.15.120:0/214071414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f287c032960 msgr2=0x7f287c0cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.990+0000 7f288259c640 1 --2- 172.21.15.120:0/214071414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f287c032960 0x7f287c0cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.990+0000 7f288259c640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.990+0000 7f288259c640 1 monclient: found mon.noname-b 2024-02-06T21:41:38.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.990+0000 7f288259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.990+0000 7f288259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.990+0000 7f288259c640 1 -- 172.21.15.120:0/214071414 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28780094d0 con 0x7f287c02e1f0 2024-02-06T21:41:38.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f57797fa640 1 -- 172.21.15.120:0/3775077712 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5770005ce0 con 0x7f577c06e780 2024-02-06T21:41:38.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f57797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f57797fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:38.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f57797fa640 10 monclient: dump: 2024-02-06T21:41:38.092 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.092 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.092 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.092 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.092 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.092 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.092 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.092 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.093 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.093 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f57797fa640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f57797fa640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f57797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f57797fa640 1 -- 172.21.15.120:0/3775077712 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5770005e40 con 0x7f577c06e780 2024-02-06T21:41:38.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f57797fa640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f57797fa640 1 -- 172.21.15.120:0/3775077712 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57700072f0 con 0x7f577c06e780 2024-02-06T21:41:38.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f57797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f57797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:38.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f57797fa640 10 monclient: dump: 2024-02-06T21:41:38.093 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.093 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.093 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.094 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.094 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.094 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.094 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.094 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.094 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.094 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f578304a640 5 monclient: authenticate success, global_id 4297 2024-02-06T21:41:38.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f578304a640 10 monclient: _renew_subs 2024-02-06T21:41:38.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f578304a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f578304a640 1 -- 172.21.15.120:0/3775077712 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577c139e30 con 0x7f577c06e780 2024-02-06T21:41:38.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f577affd640 4 set_mon_vals no callback set 2024-02-06T21:41:38.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f578304a640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f578304a640 10 monclient: _renew_subs 2024-02-06T21:41:38.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f578304a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f578304a640 1 -- 172.21.15.120:0/3775077712 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577c146b90 con 0x7f577c06e780 2024-02-06T21:41:38.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.991+0000 7f578304a640 1 librados: init done 2024-02-06T21:41:38.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.992+0000 7f578304a640 10 client.4297.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.992+0000 7f578304a640 15 client.4297.objecter _session_command_op_assign -1 1 2024-02-06T21:41:38.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.992+0000 7f578304a640 20 client.4297.objecter _get_session osd=-1 returning homeless 2024-02-06T21:41:38.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.992+0000 7f578304a640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.992+0000 7f578304a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.992+0000 7f578304a640 1 -- 172.21.15.120:0/3775077712 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f577c14ae60 con 0x7f577c06e780 2024-02-06T21:41:38.096 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.993+0000 7f0ba1818640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.096 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.993+0000 7f0ba1818640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.096 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.993+0000 7f0ba1818640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.096 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f0ba1017640 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2024-02-06T21:41:38.096 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f0ba1017640 1 -- 172.21.15.120:0/734218230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b9c06ca30 msgr2=0x7f0b9c0712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.096 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f0ba1017640 1 --2- 172.21.15.120:0/734218230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b9c06ca30 0x7f0b9c0712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.096 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f0ba1017640 1 -- 172.21.15.120:0/734218230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b9c0a9ce0 msgr2=0x7f0b9c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.096 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f0ba1017640 1 --2- 172.21.15.120:0/734218230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b9c0a9ce0 0x7f0b9c0717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.097 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f153bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.097 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f153bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.097 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f153bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f0ba1017640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f0ba1017640 1 monclient: found mon.noname-a 2024-02-06T21:41:38.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f0ba1017640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f0ba1017640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f0ba1017640 1 -- 172.21.15.120:0/734218230 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b94009b30 con 0x7f0b9c06b590 2024-02-06T21:41:38.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f2872ffd640 10 client.?.objecter ms_handle_connect 0x7f287c02e1f0 2024-02-06T21:41:38.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f2872ffd640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:38.099 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.994+0000 7f288259c640 1 --2- 172.21.15.120:0/214071414 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f287c02e1f0 0x7f287c0caf40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2878000c00 tx=0x7f2878004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.099 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.998+0000 7f57a2cda640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57940cf6e0 0x7f57940cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.100 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e0ff9640 1 -- 172.21.15.120:0/2285512625 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4068190 con 0x7fd6dc02eae0 2024-02-06T21:41:38.100 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.100 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:38.100 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e0ff9640 10 monclient: dump: 2024-02-06T21:41:38.100 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.100 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.100 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.100 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.100 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.100 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.100 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.100 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.100 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.101 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.101 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e0ff9640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.101 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e0ff9640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.101 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.101 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e0ff9640 1 -- 172.21.15.120:0/2285512625 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e404ef90 con 0x7fd6dc02eae0 2024-02-06T21:41:38.101 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.101 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e0ff9640 1 -- 172.21.15.120:0/2285512625 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4093400 con 0x7fd6dc02eae0 2024-02-06T21:41:38.101 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.101 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:38.101 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e0ff9640 10 monclient: dump: 2024-02-06T21:41:38.101 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.101 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.101 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.102 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.102 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.102 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.102 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.102 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.102 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.102 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.102 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e91bc640 5 monclient: authenticate success, global_id 4299 2024-02-06T21:41:38.102 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e91bc640 10 monclient: _renew_subs 2024-02-06T21:41:38.102 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e91bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.102 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e91bc640 1 -- 172.21.15.120:0/2285512625 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6dc0c6f60 con 0x7fd6dc02eae0 2024-02-06T21:41:38.102 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e27fc640 4 set_mon_vals no callback set 2024-02-06T21:41:38.102 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e91bc640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.102 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e91bc640 10 monclient: _renew_subs 2024-02-06T21:41:38.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e91bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e91bc640 1 -- 172.21.15.120:0/2285512625 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6dc161210 con 0x7fd6dc02eae0 2024-02-06T21:41:38.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7fd6e91bc640 1 librados: init done 2024-02-06T21:41:38.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7f57a2cda640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57940cf6e0 0x7f57940cb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38612/0 (socket says 172.21.15.120:38612) 2024-02-06T21:41:38.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7f57a2cda640 1 -- 172.21.15.120:0/3893330556 learned_addr learned my addr 172.21.15.120:0/3893330556 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7f57a2cda640 10 monclient(hunting): get_auth_request con 0x7f57940cf6e0 auth_method 0 2024-02-06T21:41:38.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7f57a2cda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7f57a2cda640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:37.999+0000 7f57a2cda640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.000+0000 7fd6e91bc640 10 client.4299.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.000+0000 7fd6e91bc640 15 client.4299.objecter _session_command_op_assign -1 1 2024-02-06T21:41:38.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.000+0000 7fd6e91bc640 20 client.4299.objecter _get_session osd=-1 returning homeless 2024-02-06T21:41:38.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.000+0000 7fd6e91bc640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.000+0000 7fd6e91bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.000+0000 7fd6e91bc640 1 -- 172.21.15.120:0/2285512625 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd6dc037610 con 0x7fd6dc02eae0 2024-02-06T21:41:38.105 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.000+0000 7f0ba1017640 1 --2- 172.21.15.120:0/734218230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b9c06b590 0x7f0b9c06deb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0b94009e50 tx=0x7f0b94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.105 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.000+0000 7f0b91ffb640 10 client.?.objecter ms_handle_connect 0x7f0b9c06b590 2024-02-06T21:41:38.105 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.000+0000 7f0b91ffb640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:38.105 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.000+0000 7f038a7e2640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f037c0a3860 0x7f037c15d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.002+0000 7f153bfff640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-02-06T21:41:38.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.002+0000 7f153bfff640 1 -- 172.21.15.120:0/3429897340 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f153c0b5170 msgr2=0x7f153c138b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.002+0000 7f153bfff640 1 --2- 172.21.15.120:0/3429897340 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f153c0b5170 0x7f153c138b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.002+0000 7f153bfff640 1 -- 172.21.15.120:0/3429897340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f153c0a97c0 msgr2=0x7f153c0b7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.002+0000 7f153bfff640 1 --2- 172.21.15.120:0/3429897340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f153c0a97c0 0x7f153c0b7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.002+0000 7f153bfff640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.002+0000 7f153bfff640 1 monclient: found mon.noname-c 2024-02-06T21:41:38.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.002+0000 7f153bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.002+0000 7f153bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.002+0000 7f153bfff640 1 -- 172.21.15.120:0/3429897340 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1534009a70 con 0x7f153c13ff70 2024-02-06T21:41:38.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.005+0000 7f153bfff640 1 --2- 172.21.15.120:0/3429897340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f153c13ff70 0x7f153c139bf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1534037040 tx=0x7f1534004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.005+0000 7f1523fff640 10 client.?.objecter ms_handle_connect 0x7f153c13ff70 2024-02-06T21:41:38.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.005+0000 7f1523fff640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:38.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.007+0000 7f57a1cd8640 1 --2- 172.21.15.120:0/3893330556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57940c8f60 0x7f57940c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.009+0000 7f57a24d9640 1 --2- 172.21.15.120:0/3893330556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5794007440 0x7f57940ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.108 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.011+0000 7f03897e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f037c032110 0x7f037c0a3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.108 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.012+0000 7f57a24d9640 10 monclient(hunting): get_auth_request con 0x7f5794007440 auth_method 0 2024-02-06T21:41:38.108 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.012+0000 7f57a24d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.108 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.012+0000 7f57a24d9640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.108 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.012+0000 7f57a24d9640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.109 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.012+0000 7f03897e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f037c032110 0x7f037c0a3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60184/0 (socket says 172.21.15.120:60184) 2024-02-06T21:41:38.109 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.012+0000 7f03897e0640 1 -- 172.21.15.120:0/132820862 learned_addr learned my addr 172.21.15.120:0/132820862 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.109 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.012+0000 7f03897e0640 10 monclient(hunting): get_auth_request con 0x7f037c032110 auth_method 0 2024-02-06T21:41:38.109 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.012+0000 7f03897e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.109 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.012+0000 7f03897e0640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.109 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.012+0000 7f03897e0640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.109 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.014+0000 7f57a24d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.109 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.014+0000 7f57a24d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.109 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.014+0000 7f57a24d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.110 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.014+0000 7f03897e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.110 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.014+0000 7f03897e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.110 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.014+0000 7f03897e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.110 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.017+0000 7f57a2cda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.110 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.017+0000 7f57a2cda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.110 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.017+0000 7f57a2cda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.111 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.017+0000 7f038a7e2640 10 monclient(hunting): get_auth_request con 0x7f037c0a3860 auth_method 0 2024-02-06T21:41:38.111 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.017+0000 7f038a7e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.111 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.017+0000 7f038a7e2640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.111 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.017+0000 7f038a7e2640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.112 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.018+0000 7f2872ffd640 1 -- 172.21.15.120:0/214071414 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2878039c90 con 0x7f287c02e1f0 2024-02-06T21:41:38.112 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.018+0000 7f2872ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.113 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.018+0000 7f2872ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:38.113 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.018+0000 7f2872ffd640 10 monclient: dump: 2024-02-06T21:41:38.113 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.113 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.113 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.113 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.113 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.113 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.113 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.113 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.113 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.113 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.113 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.018+0000 7f2872ffd640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.113 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.018+0000 7f2872ffd640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.113 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.018+0000 7f2872ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.018+0000 7f2872ffd640 1 -- 172.21.15.120:0/214071414 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2878039df0 con 0x7f287c02e1f0 2024-02-06T21:41:38.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.018+0000 7f2872ffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.018+0000 7f2872ffd640 1 -- 172.21.15.120:0/214071414 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2878043640 con 0x7f287c02e1f0 2024-02-06T21:41:38.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.018+0000 7f2872ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.018+0000 7f2872ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:38.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.018+0000 7f2872ffd640 10 monclient: dump: 2024-02-06T21:41:38.114 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.114 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.114 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.114 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.114 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.114 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.114 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.114 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.115 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.115 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.019+0000 7f288359e640 5 monclient: authenticate success, global_id 4306 2024-02-06T21:41:38.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.018+0000 7f2880d99640 4 set_mon_vals no callback set 2024-02-06T21:41:38.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.019+0000 7f288359e640 10 monclient: _renew_subs 2024-02-06T21:41:38.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.019+0000 7f288359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.019+0000 7f288359e640 1 -- 172.21.15.120:0/214071414 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f287c0a3340 con 0x7f287c02e1f0 2024-02-06T21:41:38.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.019+0000 7f288359e640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.019+0000 7f288359e640 10 monclient: _renew_subs 2024-02-06T21:41:38.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.019+0000 7f288359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.019+0000 7f288359e640 1 -- 172.21.15.120:0/214071414 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287c0cc160 con 0x7f287c02e1f0 2024-02-06T21:41:38.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.019+0000 7f288359e640 1 librados: init done 2024-02-06T21:41:38.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.019+0000 7f288359e640 10 client.4306.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.019+0000 7f288359e640 15 client.4306.objecter _session_command_op_assign -1 1 2024-02-06T21:41:38.116 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.019+0000 7f288359e640 20 client.4306.objecter _get_session osd=-1 returning homeless 2024-02-06T21:41:38.116 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.019+0000 7f288359e640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.116 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.019+0000 7f288359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.116 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.019+0000 7f288359e640 1 -- 172.21.15.120:0/214071414 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f287c0cd1c0 con 0x7f287c02e1f0 2024-02-06T21:41:38.116 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.020+0000 7f038a7e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.116 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.020+0000 7f038a7e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.116 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.020+0000 7f038a7e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.023+0000 7f57a2cda640 10 monclient(hunting): handle_auth_done global_id 4312 payload 824 2024-02-06T21:41:38.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.023+0000 7f57a2cda640 1 -- 172.21.15.120:0/3893330556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57940c8f60 msgr2=0x7f57940c9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.023+0000 7f57a2cda640 1 --2- 172.21.15.120:0/3893330556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57940c8f60 0x7f57940c9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.023+0000 7f57a2cda640 1 -- 172.21.15.120:0/3893330556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5794007440 msgr2=0x7f57940ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.023+0000 7f57a2cda640 1 --2- 172.21.15.120:0/3893330556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5794007440 0x7f57940ce7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.023+0000 7f57a2cda640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.023+0000 7f57a2cda640 1 monclient: found mon.noname-b 2024-02-06T21:41:38.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.023+0000 7f57a2cda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.023+0000 7f57a2cda640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.023+0000 7f57a2cda640 1 -- 172.21.15.120:0/3893330556 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579c073ad0 con 0x7f57940cf6e0 2024-02-06T21:41:38.118 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.025+0000 7f03897e0640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-02-06T21:41:38.118 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.025+0000 7f03897e0640 1 -- 172.21.15.120:0/132820862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c007190 msgr2=0x7f037c034c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.118 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.025+0000 7f03897e0640 1 --2- 172.21.15.120:0/132820862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c007190 0x7f037c034c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.025+0000 7f03897e0640 1 -- 172.21.15.120:0/132820862 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f037c0a3860 msgr2=0x7f037c15d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.025+0000 7f03897e0640 1 --2- 172.21.15.120:0/132820862 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f037c0a3860 0x7f037c15d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.025+0000 7f03897e0640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.025+0000 7f03897e0640 1 monclient: found mon.noname-a 2024-02-06T21:41:38.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.025+0000 7f03897e0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.025+0000 7f03897e0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.025+0000 7f03897e0640 1 -- 172.21.15.120:0/132820862 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c15dc40 con 0x7f037c032110 2024-02-06T21:41:38.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.026+0000 7fdf1a7fc640 1 -- 172.21.15.120:0/403538856 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdf1c051050 con 0x7fdf2413ba80 2024-02-06T21:41:38.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7fdf1a7fc640 1 --2- 172.21.15.120:0/403538856 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fdf0806d220 0x7fdf0806f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7fdf299ff640 1 --2- 172.21.15.120:0/403538856 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fdf0806d220 0x7fdf0806f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7fdf299ff640 10 monclient: get_auth_request con 0x7fdf0806d220 auth_method 0 2024-02-06T21:41:38.121 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7f0b91ffb640 1 -- 172.21.15.120:0/734218230 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9403e450 con 0x7f0b9c06b590 2024-02-06T21:41:38.121 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7f0b91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.121 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7f0b91ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:38.121 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7f0b91ffb640 10 monclient: dump: 2024-02-06T21:41:38.121 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.121 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.121 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.122 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.122 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.122 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.122 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.122 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.122 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.122 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.122 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7f0b91ffb640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.122 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7f0b91ffb640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.122 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7f0b91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.122 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7f0b91ffb640 1 -- 172.21.15.120:0/734218230 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9403e5b0 con 0x7f0b9c06b590 2024-02-06T21:41:38.122 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7f0b91ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.122 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.028+0000 7fdf299ff640 1 --2- 172.21.15.120:0/403538856 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fdf0806d220 0x7fdf0806f6e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdf14006fd0 tx=0x7fdf14008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.028+0000 7fdf1a7fc640 10 client.4274.objecter ms_handle_connect 0x7fdf0806d220 2024-02-06T21:41:38.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7f0b91ffb640 1 -- 172.21.15.120:0/734218230 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9403e960 con 0x7f0b9c06b590 2024-02-06T21:41:38.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7f0b91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7f0b91ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:38.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7f0b91ffb640 10 monclient: dump: 2024-02-06T21:41:38.125 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.125 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.125 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.125 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.125 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.125 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.125 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.125 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.125 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.125 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.027+0000 7f0b937fe640 4 set_mon_vals no callback set 2024-02-06T21:41:38.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.028+0000 7f0ba327b640 5 monclient: authenticate success, global_id 4302 2024-02-06T21:41:38.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.028+0000 7f0ba327b640 10 monclient: _renew_subs 2024-02-06T21:41:38.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.028+0000 7f0ba327b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.028+0000 7f0ba327b640 1 -- 172.21.15.120:0/734218230 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9c06fb70 con 0x7f0b9c06b590 2024-02-06T21:41:38.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.028+0000 7f0ba327b640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.028+0000 7f0ba327b640 10 monclient: _renew_subs 2024-02-06T21:41:38.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.028+0000 7f0ba327b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.028+0000 7f0ba327b640 1 -- 172.21.15.120:0/734218230 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9c071d30 con 0x7f0b9c06b590 2024-02-06T21:41:38.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.028+0000 7f0ba327b640 1 librados: init done 2024-02-06T21:41:38.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.029+0000 7f0ba327b640 10 client.4302.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.029+0000 7f0ba327b640 15 client.4302.objecter _session_command_op_assign -1 1 2024-02-06T21:41:38.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.029+0000 7f0ba327b640 20 client.4302.objecter _get_session osd=-1 returning homeless 2024-02-06T21:41:38.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.029+0000 7f0ba327b640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.029+0000 7f0ba327b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.029+0000 7f0ba327b640 1 -- 172.21.15.120:0/734218230 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0b9c0af3c0 con 0x7f0b9c06b590 2024-02-06T21:41:38.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 1 -- 172.21.15.120:0/403538856 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdf1c07b5e0 con 0x7fdf2413ba80 2024-02-06T21:41:38.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 10 client.4274.objecter ms_dispatch 0x7fdf24000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:38.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 3 client.4274.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:38.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 3 client.4274.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:38.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 10 client.4274.objecter checking command 1 2024-02-06T21:41:38.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 1 --2- 172.21.15.120:0/403538856 >> [v2:172.21.15.120:6800/3324566974,v1:172.21.15.120:6801/3324566974] conn(0x7fdf08073660 0x7fdf08075a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 20 client.4274.objecter _get_session s=0x7fdf08072f90 osd=3 3 2024-02-06T21:41:38.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 20 client.4274.objecter put_session s=0x7fdf08072f90 osd=3 3 2024-02-06T21:41:38.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 15 client.4274.objecter _session_command_op_remove -1 1 2024-02-06T21:41:38.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 20 client.4274.objecter _get_session s=0x7fdf08072f90 osd=3 3 2024-02-06T21:41:38.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 20 client.4274.objecter get_session s=0x7fdf08072f90 osd=3 3 2024-02-06T21:41:38.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 15 client.4274.objecter _session_command_op_assign 3 1 2024-02-06T21:41:38.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 20 client.4274.objecter put_session s=0x7fdf08072f90 osd=3 4 2024-02-06T21:41:38.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 10 client.4274.objecter _send_command 1 2024-02-06T21:41:38.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 1 -- 172.21.15.120:0/403538856 --> [v2:172.21.15.120:6800/3324566974,v1:172.21.15.120:6801/3324566974] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf08076130 con 0x7fdf08073660 2024-02-06T21:41:38.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 20 client.4274.objecter dump_active .. 0 homeless 2024-02-06T21:41:38.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 1 -- 172.21.15.120:0/403538856 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf1c0072e0 con 0x7fdf2413ba80 2024-02-06T21:41:38.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.032+0000 7fdf1a7fc640 10 monclient: handle_get_version_reply finishing 1 version 23 2024-02-06T21:41:38.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.033+0000 7fdf291fe640 1 --2- 172.21.15.120:0/403538856 >> [v2:172.21.15.120:6800/3324566974,v1:172.21.15.120:6801/3324566974] conn(0x7fdf08073660 0x7fdf08075a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.033+0000 7fdf291fe640 10 monclient: get_auth_request con 0x7fdf08073660 auth_method 0 2024-02-06T21:41:38.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1523fff640 1 -- 172.21.15.120:0/3429897340 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153403e610 con 0x7f153c13ff70 2024-02-06T21:41:38.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1523fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1523fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:38.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1523fff640 10 monclient: dump: 2024-02-06T21:41:38.130 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.130 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.130 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.130 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.130 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.131 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.131 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.131 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.131 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.131 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1523fff640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1523fff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1523fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1523fff640 1 -- 172.21.15.120:0/3429897340 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153403e770 con 0x7f153c13ff70 2024-02-06T21:41:38.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1523fff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1523fff640 1 -- 172.21.15.120:0/3429897340 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153403eb00 con 0x7f153c13ff70 2024-02-06T21:41:38.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1523fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1523fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:38.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1523fff640 10 monclient: dump: 2024-02-06T21:41:38.132 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.132 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.132 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.132 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.132 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.132 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.132 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.132 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.132 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.132 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1540899640 5 monclient: authenticate success, global_id 4280 2024-02-06T21:41:38.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1540899640 10 monclient: _renew_subs 2024-02-06T21:41:38.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1540899640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1540899640 1 -- 172.21.15.120:0/3429897340 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153c13aec0 con 0x7f153c13ff70 2024-02-06T21:41:38.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1539ffb640 4 set_mon_vals no callback set 2024-02-06T21:41:38.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1540899640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1540899640 10 monclient: _renew_subs 2024-02-06T21:41:38.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1540899640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1540899640 1 -- 172.21.15.120:0/3429897340 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153c0abf10 con 0x7f153c13ff70 2024-02-06T21:41:38.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.035+0000 7f1540899640 1 librados: init done 2024-02-06T21:41:38.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.036+0000 7f1540899640 10 client.4280.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.036+0000 7f1540899640 15 client.4280.objecter _session_command_op_assign -1 1 2024-02-06T21:41:38.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.036+0000 7f1540899640 20 client.4280.objecter _get_session osd=-1 returning homeless 2024-02-06T21:41:38.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.036+0000 7f1540899640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.036+0000 7f1540899640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.036+0000 7f1540899640 1 -- 172.21.15.120:0/3429897340 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f153c0b31c0 con 0x7f153c13ff70 2024-02-06T21:41:38.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.037+0000 7fdf291fe640 1 --2- 172.21.15.120:0/403538856 >> [v2:172.21.15.120:6800/3324566974,v1:172.21.15.120:6801/3324566974] conn(0x7fdf08073660 0x7fdf08075a80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.037+0000 7fdf1a7fc640 10 client.4274.objecter ms_handle_connect 0x7fdf08073660 2024-02-06T21:41:38.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.037+0000 7f03897e0640 1 --2- 172.21.15.120:0/132820862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f037c032110 0x7f037c0a3320 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0374007f90 tx=0x7f0374001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.037+0000 7f037a7fc640 10 client.?.objecter ms_handle_connect 0x7f037c032110 2024-02-06T21:41:38.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.037+0000 7f037a7fc640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:38.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.053+0000 7f2872ffd640 1 -- 172.21.15.120:0/214071414 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f28780437a0 con 0x7f287c02e1f0 2024-02-06T21:41:38.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.053+0000 7f2872ffd640 1 --2- 172.21.15.120:0/214071414 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f286406d1f0 0x7f286406f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.053+0000 7f2881d9b640 1 --2- 172.21.15.120:0/214071414 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f286406d1f0 0x7f286406f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.054+0000 7f2881d9b640 10 monclient: get_auth_request con 0x7f286406d1f0 auth_method 0 2024-02-06T21:41:38.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.054+0000 7f2881d9b640 1 --2- 172.21.15.120:0/214071414 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f286406d1f0 0x7f286406f6b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2884068c70 tx=0x7f288406e040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.054+0000 7f2872ffd640 10 client.4306.objecter ms_handle_connect 0x7f286406d1f0 2024-02-06T21:41:38.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.054+0000 7f0b91ffb640 1 -- 172.21.15.120:0/734218230 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0b9402cd50 con 0x7f0b9c06b590 2024-02-06T21:41:38.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.054+0000 7f0b91ffb640 1 --2- 172.21.15.120:0/734218230 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f0b8006d1f0 0x7f0b8006f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.055+0000 7f0ba0816640 1 --2- 172.21.15.120:0/734218230 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f0b8006d1f0 0x7f0b8006f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.055+0000 7f0ba0816640 10 monclient: get_auth_request con 0x7f0b8006d1f0 auth_method 0 2024-02-06T21:41:38.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.055+0000 7f0b91ffb640 10 client.4302.objecter ms_handle_connect 0x7f0b8006d1f0 2024-02-06T21:41:38.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.055+0000 7f0ba0816640 1 --2- 172.21.15.120:0/734218230 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f0b8006d1f0 0x7f0b8006f6b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c006fd0 tx=0x7f0b8c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.057+0000 7f2872ffd640 1 -- 172.21.15.120:0/214071414 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f287803e070 con 0x7f287c02e1f0 2024-02-06T21:41:38.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.057+0000 7f2872ffd640 10 client.4306.objecter ms_dispatch 0x7f2884000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:38.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.057+0000 7f2872ffd640 3 client.4306.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:38.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.057+0000 7f2872ffd640 3 client.4306.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:38.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.058+0000 7f2872ffd640 10 client.4306.objecter checking command 1 2024-02-06T21:41:38.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.058+0000 7f2872ffd640 1 --2- 172.21.15.120:0/214071414 >> [v2:172.21.15.135:6802/2191547918,v1:172.21.15.135:6803/2191547918] conn(0x7f2864073630 0x7f2864075a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.058+0000 7f2872ffd640 20 client.4306.objecter _get_session s=0x7f2864072f60 osd=4 3 2024-02-06T21:41:38.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.058+0000 7f2872ffd640 20 client.4306.objecter put_session s=0x7f2864072f60 osd=4 3 2024-02-06T21:41:38.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.058+0000 7f2872ffd640 15 client.4306.objecter _session_command_op_remove -1 1 2024-02-06T21:41:38.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.058+0000 7f2872ffd640 20 client.4306.objecter _get_session s=0x7f2864072f60 osd=4 3 2024-02-06T21:41:38.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.058+0000 7f2872ffd640 20 client.4306.objecter get_session s=0x7f2864072f60 osd=4 3 2024-02-06T21:41:38.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.058+0000 7f2872ffd640 15 client.4306.objecter _session_command_op_assign 4 1 2024-02-06T21:41:38.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.058+0000 7f2872ffd640 20 client.4306.objecter put_session s=0x7f2864072f60 osd=4 4 2024-02-06T21:41:38.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.058+0000 7f2872ffd640 10 client.4306.objecter _send_command 1 2024-02-06T21:41:38.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.058+0000 7f2872ffd640 1 -- 172.21.15.120:0/214071414 --> [v2:172.21.15.135:6802/2191547918,v1:172.21.15.135:6803/2191547918] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2864076100 con 0x7f2864073630 2024-02-06T21:41:38.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.058+0000 7f2872ffd640 20 client.4306.objecter dump_active .. 0 homeless 2024-02-06T21:41:38.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.058+0000 7f2882d9d640 1 --2- 172.21.15.120:0/214071414 >> [v2:172.21.15.135:6802/2191547918,v1:172.21.15.135:6803/2191547918] conn(0x7f2864073630 0x7f2864075a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.059+0000 7f2882d9d640 10 monclient: get_auth_request con 0x7f2864073630 auth_method 0 2024-02-06T21:41:38.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.059+0000 7f2872ffd640 1 -- 172.21.15.120:0/214071414 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2878078640 con 0x7f287c02e1f0 2024-02-06T21:41:38.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.059+0000 7f2872ffd640 10 monclient: handle_get_version_reply finishing 1 version 23 2024-02-06T21:41:38.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 1 -- 172.21.15.120:0/734218230 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0b9407c1a0 con 0x7f0b9c06b590 2024-02-06T21:41:38.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 10 client.4302.objecter ms_dispatch 0x7f0b9c000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:38.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 3 client.4302.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:38.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 3 client.4302.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:38.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 10 client.4302.objecter checking command 1 2024-02-06T21:41:38.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 1 --2- 172.21.15.120:0/734218230 >> [v2:172.21.15.135:6825/3740758337,v1:172.21.15.135:6827/3740758337] conn(0x7f0b80073630 0x7f0b80075a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 20 client.4302.objecter _get_session s=0x7f0b80072f60 osd=7 3 2024-02-06T21:41:38.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 20 client.4302.objecter put_session s=0x7f0b80072f60 osd=7 3 2024-02-06T21:41:38.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 15 client.4302.objecter _session_command_op_remove -1 1 2024-02-06T21:41:38.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 20 client.4302.objecter _get_session s=0x7f0b80072f60 osd=7 3 2024-02-06T21:41:38.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 20 client.4302.objecter get_session s=0x7f0b80072f60 osd=7 3 2024-02-06T21:41:38.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 15 client.4302.objecter _session_command_op_assign 7 1 2024-02-06T21:41:38.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 20 client.4302.objecter put_session s=0x7f0b80072f60 osd=7 4 2024-02-06T21:41:38.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 10 client.4302.objecter _send_command 1 2024-02-06T21:41:38.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 1 -- 172.21.15.120:0/734218230 --> [v2:172.21.15.135:6825/3740758337,v1:172.21.15.135:6827/3740758337] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b80076100 con 0x7f0b80073630 2024-02-06T21:41:38.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 20 client.4302.objecter dump_active .. 0 homeless 2024-02-06T21:41:38.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 1 -- 172.21.15.120:0/734218230 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b94077d40 con 0x7f0b9c06b590 2024-02-06T21:41:38.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.060+0000 7f0b91ffb640 10 monclient: handle_get_version_reply finishing 1 version 23 2024-02-06T21:41:38.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.061+0000 7f1523fff640 1 -- 172.21.15.120:0/3429897340 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f153403b050 con 0x7f153c13ff70 2024-02-06T21:41:38.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.061+0000 7f1523fff640 1 --2- 172.21.15.120:0/3429897340 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f152806d1f0 0x7f152806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.062+0000 7f0ba1818640 1 --2- 172.21.15.120:0/734218230 >> [v2:172.21.15.135:6825/3740758337,v1:172.21.15.135:6827/3740758337] conn(0x7f0b80073630 0x7f0b80075a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.062+0000 7f153b7fe640 1 --2- 172.21.15.120:0/3429897340 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f152806d1f0 0x7f152806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.062+0000 7f153b7fe640 10 monclient: get_auth_request con 0x7f152806d1f0 auth_method 0 2024-02-06T21:41:38.143 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.062+0000 7f0ba1818640 10 monclient: get_auth_request con 0x7f0b80073630 auth_method 0 2024-02-06T21:41:38.143 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.062+0000 7f1523fff640 10 client.4280.objecter ms_handle_connect 0x7f152806d1f0 2024-02-06T21:41:38.143 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.062+0000 7f153b7fe640 1 --2- 172.21.15.120:0/3429897340 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f152806d1f0 0x7f152806f6b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f152c0015f0 tx=0x7f152c007a40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.143 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.063+0000 7f2872ffd640 10 client.4306.objecter ms_handle_connect 0x7f2864073630 2024-02-06T21:41:38.144 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.063+0000 7f2882d9d640 1 --2- 172.21.15.120:0/214071414 >> [v2:172.21.15.135:6802/2191547918,v1:172.21.15.135:6803/2191547918] conn(0x7f2864073630 0x7f2864075a50 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.144 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 1 -- 172.21.15.120:0/3429897340 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f153407c000 con 0x7f153c13ff70 2024-02-06T21:41:38.144 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 10 client.4280.objecter ms_dispatch 0x7f153c000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:38.144 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 3 client.4280.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:38.144 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 3 client.4280.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:38.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 10 client.4280.objecter checking command 1 2024-02-06T21:41:38.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 1 --2- 172.21.15.120:0/3429897340 >> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] conn(0x7f1528073630 0x7f1528075a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 20 client.4280.objecter _get_session s=0x7f1528072f60 osd=1 3 2024-02-06T21:41:38.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 20 client.4280.objecter put_session s=0x7f1528072f60 osd=1 3 2024-02-06T21:41:38.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 15 client.4280.objecter _session_command_op_remove -1 1 2024-02-06T21:41:38.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 20 client.4280.objecter _get_session s=0x7f1528072f60 osd=1 3 2024-02-06T21:41:38.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 20 client.4280.objecter get_session s=0x7f1528072f60 osd=1 3 2024-02-06T21:41:38.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 15 client.4280.objecter _session_command_op_assign 1 1 2024-02-06T21:41:38.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 20 client.4280.objecter put_session s=0x7f1528072f60 osd=1 4 2024-02-06T21:41:38.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 10 client.4280.objecter _send_command 1 2024-02-06T21:41:38.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 1 -- 172.21.15.120:0/3429897340 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1528076100 con 0x7f1528073630 2024-02-06T21:41:38.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 20 client.4280.objecter dump_active .. 0 homeless 2024-02-06T21:41:38.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 1 -- 172.21.15.120:0/3429897340 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1534077ba0 con 0x7f153c13ff70 2024-02-06T21:41:38.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f1523fff640 10 monclient: handle_get_version_reply finishing 1 version 23 2024-02-06T21:41:38.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.065+0000 7f153affd640 1 --2- 172.21.15.120:0/3429897340 >> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] conn(0x7f1528073630 0x7f1528075a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.066+0000 7f153affd640 10 monclient: get_auth_request con 0x7f1528073630 auth_method 0 2024-02-06T21:41:38.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.066+0000 7f0b91ffb640 10 client.4302.objecter ms_handle_connect 0x7f0b80073630 2024-02-06T21:41:38.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.066+0000 7f0ba1818640 1 --2- 172.21.15.120:0/734218230 >> [v2:172.21.15.135:6825/3740758337,v1:172.21.15.135:6827/3740758337] conn(0x7f0b80073630 0x7f0b80075a50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.069+0000 7f1523fff640 10 client.4280.objecter ms_handle_connect 0x7f1528073630 2024-02-06T21:41:38.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.069+0000 7f153affd640 1 --2- 172.21.15.120:0/3429897340 >> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] conn(0x7f1528073630 0x7f1528075a50 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.075+0000 7fd6e0ff9640 1 -- 172.21.15.120:0/2285512625 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd6e4093560 con 0x7fd6dc02eae0 2024-02-06T21:41:38.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.075+0000 7fd6e0ff9640 1 --2- 172.21.15.120:0/2285512625 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd6c806d1f0 0x7fd6c806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.076+0000 7fd6e3fff640 1 --2- 172.21.15.120:0/2285512625 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd6c806d1f0 0x7fd6c806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.076+0000 7fd6e3fff640 10 monclient: get_auth_request con 0x7fd6c806d1f0 auth_method 0 2024-02-06T21:41:38.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.076+0000 7fd6e0ff9640 10 client.4299.objecter ms_handle_connect 0x7fd6c806d1f0 2024-02-06T21:41:38.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.076+0000 7fd6e3fff640 1 --2- 172.21.15.120:0/2285512625 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd6c806d1f0 0x7fd6c806f6b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8034000 tx=0x7fd6d8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.079+0000 7f57a2cda640 1 --2- 172.21.15.120:0/3893330556 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57940cf6e0 0x7f57940cb870 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f579c02d340 tx=0x7f579c068b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.079+0000 7f5792ffd640 10 client.?.objecter ms_handle_connect 0x7f57940cf6e0 2024-02-06T21:41:38.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.079+0000 7f5792ffd640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:38.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.079+0000 7f57797fa640 1 -- 172.21.15.120:0/3775077712 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5770007450 con 0x7f577c06e780 2024-02-06T21:41:38.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.080+0000 7f57797fa640 1 --2- 172.21.15.120:0/3775077712 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f576806d1a0 0x7f576806f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.080+0000 7f577bfff640 1 --2- 172.21.15.120:0/3775077712 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f576806d1a0 0x7f576806f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.080+0000 7f577bfff640 10 monclient: get_auth_request con 0x7f576806d1a0 auth_method 0 2024-02-06T21:41:38.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.080+0000 7f57797fa640 10 client.4297.objecter ms_handle_connect 0x7f576806d1a0 2024-02-06T21:41:38.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.080+0000 7f577bfff640 1 --2- 172.21.15.120:0/3775077712 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f576806d1a0 0x7f576806f660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f576c036000 tx=0x7f576c03d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.150 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.080+0000 7fd6e0ff9640 1 -- 172.21.15.120:0/2285512625 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd6e40d3c00 con 0x7fd6dc02eae0 2024-02-06T21:41:38.150 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.080+0000 7fd6e0ff9640 10 client.4299.objecter ms_dispatch 0x7fd6e4000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:38.150 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.080+0000 7fd6e0ff9640 3 client.4299.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:38.150 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.080+0000 7fd6e0ff9640 3 client.4299.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:38.150 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.081+0000 7fd6e0ff9640 10 client.4299.objecter checking command 1 2024-02-06T21:41:38.150 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.081+0000 7fd6e0ff9640 1 --2- 172.21.15.120:0/2285512625 >> [v2:172.21.15.135:6810/1080459988,v1:172.21.15.135:6811/1080459988] conn(0x7fd6c8073630 0x7fd6c8075a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.150 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.081+0000 7fd6e0ff9640 20 client.4299.objecter _get_session s=0x7fd6c8072f60 osd=6 3 2024-02-06T21:41:38.150 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.081+0000 7fd6e0ff9640 20 client.4299.objecter put_session s=0x7fd6c8072f60 osd=6 3 2024-02-06T21:41:38.150 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.081+0000 7fd6e0ff9640 15 client.4299.objecter _session_command_op_remove -1 1 2024-02-06T21:41:38.150 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.081+0000 7fd6e0ff9640 20 client.4299.objecter _get_session s=0x7fd6c8072f60 osd=6 3 2024-02-06T21:41:38.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.081+0000 7fd6e0ff9640 20 client.4299.objecter get_session s=0x7fd6c8072f60 osd=6 3 2024-02-06T21:41:38.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.081+0000 7fd6e0ff9640 15 client.4299.objecter _session_command_op_assign 6 1 2024-02-06T21:41:38.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.081+0000 7fd6e0ff9640 20 client.4299.objecter put_session s=0x7fd6c8072f60 osd=6 4 2024-02-06T21:41:38.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.081+0000 7fd6e0ff9640 10 client.4299.objecter _send_command 1 2024-02-06T21:41:38.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.081+0000 7fd6e0ff9640 1 -- 172.21.15.120:0/2285512625 --> [v2:172.21.15.135:6810/1080459988,v1:172.21.15.135:6811/1080459988] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6c8076100 con 0x7fd6c8073630 2024-02-06T21:41:38.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.081+0000 7fd6e0ff9640 20 client.4299.objecter dump_active .. 0 homeless 2024-02-06T21:41:38.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.081+0000 7fd6e0ff9640 1 -- 172.21.15.120:0/2285512625 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6e40cf7a0 con 0x7fd6dc02eae0 2024-02-06T21:41:38.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.081+0000 7fd6e0ff9640 10 monclient: handle_get_version_reply finishing 1 version 23 2024-02-06T21:41:38.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.081+0000 7fd6e37fe640 1 --2- 172.21.15.120:0/2285512625 >> [v2:172.21.15.135:6810/1080459988,v1:172.21.15.135:6811/1080459988] conn(0x7fd6c8073630 0x7fd6c8075a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7fd6e37fe640 10 monclient: get_auth_request con 0x7fd6c8073630 auth_method 0 2024-02-06T21:41:38.153 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f037a7fc640 1 -- 172.21.15.120:0/132820862 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037402ca70 con 0x7f037c032110 2024-02-06T21:41:38.153 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f037a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.153 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f037a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:38.153 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f037a7fc640 10 monclient: dump: 2024-02-06T21:41:38.153 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.153 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.154 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.154 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.154 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.154 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.154 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.154 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.154 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.154 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.154 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f037a7fc640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.154 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f037a7fc640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.154 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f037a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.154 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f038c245640 5 monclient: authenticate success, global_id 4311 2024-02-06T21:41:38.154 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f038c245640 10 monclient: _renew_subs 2024-02-06T21:41:38.154 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f038c245640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.154 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f038c245640 1 -- 172.21.15.120:0/132820862 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037c15ddd0 con 0x7f037c032110 2024-02-06T21:41:38.155 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f038c245640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.155 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f038c245640 10 monclient: _renew_subs 2024-02-06T21:41:38.155 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f038c245640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.155 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f038c245640 1 -- 172.21.15.120:0/132820862 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f037c0c18a0 con 0x7f037c032110 2024-02-06T21:41:38.155 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f038c245640 1 librados: init done 2024-02-06T21:41:38.155 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f037a7fc640 1 -- 172.21.15.120:0/132820862 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037402cbf0 con 0x7f037c032110 2024-02-06T21:41:38.155 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f037a7fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.155 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f037a7fc640 1 -- 172.21.15.120:0/132820862 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0374007070 con 0x7f037c032110 2024-02-06T21:41:38.155 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f037a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.155 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f037a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:38.155 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.082+0000 7f037a7fc640 10 monclient: dump: 2024-02-06T21:41:38.155 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.155 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.155 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.155 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.156 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.156 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.156 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.156 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.156 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.156 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.156 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.083+0000 7f037bfff640 4 set_mon_vals no callback set 2024-02-06T21:41:38.156 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.083+0000 7f038c245640 10 client.4311.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.156 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.083+0000 7f038c245640 15 client.4311.objecter _session_command_op_assign -1 1 2024-02-06T21:41:38.156 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.083+0000 7f038c245640 20 client.4311.objecter _get_session osd=-1 returning homeless 2024-02-06T21:41:38.156 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.083+0000 7f038c245640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.156 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.083+0000 7f038c245640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.156 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.083+0000 7f038c245640 1 -- 172.21.15.120:0/132820862 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f036c002910 con 0x7f037c032110 2024-02-06T21:41:38.157 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 1 -- 172.21.15.120:0/3775077712 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5770058e30 con 0x7f577c06e780 2024-02-06T21:41:38.157 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 10 client.4297.objecter ms_dispatch 0x7f577c000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:38.157 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 3 client.4297.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:38.157 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 3 client.4297.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:38.157 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 10 client.4297.objecter checking command 1 2024-02-06T21:41:38.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 1 --2- 172.21.15.120:0/3775077712 >> [v2:172.21.15.120:6816/2062960776,v1:172.21.15.120:6817/2062960776] conn(0x7f57680735e0 0x7f5768075a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 20 client.4297.objecter _get_session s=0x7f5768072f10 osd=0 3 2024-02-06T21:41:38.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 20 client.4297.objecter put_session s=0x7f5768072f10 osd=0 3 2024-02-06T21:41:38.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 15 client.4297.objecter _session_command_op_remove -1 1 2024-02-06T21:41:38.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 20 client.4297.objecter _get_session s=0x7f5768072f10 osd=0 3 2024-02-06T21:41:38.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 20 client.4297.objecter get_session s=0x7f5768072f10 osd=0 3 2024-02-06T21:41:38.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 15 client.4297.objecter _session_command_op_assign 0 1 2024-02-06T21:41:38.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 20 client.4297.objecter put_session s=0x7f5768072f10 osd=0 4 2024-02-06T21:41:38.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 10 client.4297.objecter _send_command 1 2024-02-06T21:41:38.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 1 -- 172.21.15.120:0/3775077712 --> [v2:172.21.15.120:6816/2062960776,v1:172.21.15.120:6817/2062960776] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57680760b0 con 0x7f57680735e0 2024-02-06T21:41:38.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.085+0000 7f57797fa640 20 client.4297.objecter dump_active .. 0 homeless 2024-02-06T21:41:38.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.086+0000 7f57815e7640 1 --2- 172.21.15.120:0/3775077712 >> [v2:172.21.15.120:6816/2062960776,v1:172.21.15.120:6817/2062960776] conn(0x7f57680735e0 0x7f5768075a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.086+0000 7f57797fa640 1 -- 172.21.15.120:0/3775077712 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57700549d0 con 0x7f577c06e780 2024-02-06T21:41:38.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.086+0000 7f57797fa640 10 monclient: handle_get_version_reply finishing 1 version 23 2024-02-06T21:41:38.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.086+0000 7f57815e7640 10 monclient: get_auth_request con 0x7f57680735e0 auth_method 0 2024-02-06T21:41:38.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.088+0000 7fd6e37fe640 1 --2- 172.21.15.120:0/2285512625 >> [v2:172.21.15.135:6810/1080459988,v1:172.21.15.135:6811/1080459988] conn(0x7fd6c8073630 0x7fd6c8075a50 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.088+0000 7fd6e0ff9640 10 client.4299.objecter ms_handle_connect 0x7fd6c8073630 2024-02-06T21:41:38.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f57797fa640 10 client.4297.objecter ms_handle_connect 0x7f57680735e0 2024-02-06T21:41:38.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f57815e7640 1 --2- 172.21.15.120:0/3775077712 >> [v2:172.21.15.120:6816/2062960776,v1:172.21.15.120:6817/2062960776] conn(0x7f57680735e0 0x7f5768075a00 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.161 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f5792ffd640 1 -- 172.21.15.120:0/3893330556 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c0b9920 con 0x7f57940cf6e0 2024-02-06T21:41:38.161 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f5792ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.161 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f5792ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:38.161 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f5792ffd640 10 monclient: dump: 2024-02-06T21:41:38.161 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.161 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.161 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.161 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.161 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.161 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.162 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.162 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.162 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.162 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.162 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f5792ffd640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.162 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f5792ffd640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.162 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f5792ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.162 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f5792ffd640 1 -- 172.21.15.120:0/3893330556 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579c0b9a80 con 0x7f57940cf6e0 2024-02-06T21:41:38.162 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f5792ffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.162 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f5792ffd640 1 -- 172.21.15.120:0/3893330556 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c0b9df0 con 0x7f57940cf6e0 2024-02-06T21:41:38.162 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f5792ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.162 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f5792ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:38.162 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f5792ffd640 10 monclient: dump: 2024-02-06T21:41:38.162 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.162 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.163 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.163 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.163 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.163 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.163 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.163 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.163 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.163 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.163 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f57a473d640 5 monclient: authenticate success, global_id 4312 2024-02-06T21:41:38.163 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f57a473d640 10 monclient: _renew_subs 2024-02-06T21:41:38.163 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f57a0cd6640 4 set_mon_vals no callback set 2024-02-06T21:41:38.163 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f57a473d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.163 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f57a473d640 1 -- 172.21.15.120:0/3893330556 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57940cad00 con 0x7f57940cf6e0 2024-02-06T21:41:38.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f57a473d640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f57a473d640 10 monclient: _renew_subs 2024-02-06T21:41:38.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f57a473d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f57a473d640 1 -- 172.21.15.120:0/3893330556 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57940cafb0 con 0x7f57940cf6e0 2024-02-06T21:41:38.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.091+0000 7f57a473d640 1 librados: init done 2024-02-06T21:41:38.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.092+0000 7f57a473d640 10 client.4312.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.092+0000 7f57a473d640 15 client.4312.objecter _session_command_op_assign -1 1 2024-02-06T21:41:38.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.092+0000 7f57a473d640 20 client.4312.objecter _get_session osd=-1 returning homeless 2024-02-06T21:41:38.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.092+0000 7f57a473d640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.092+0000 7f57a473d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.092+0000 7f57a473d640 1 -- 172.21.15.120:0/3893330556 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f579400b6d0 con 0x7f57940cf6e0 2024-02-06T21:41:38.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.093+0000 7fdf1a7fc640 1 -- 172.21.15.120:0/403538856 <== osd.3 v2:172.21.15.120:6800/3324566974 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdf08076130 con 0x7fdf08073660 2024-02-06T21:41:38.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.093+0000 7fdf1a7fc640 10 client.4274.objecter ms_dispatch 0x7fdf24000b90 command_reply(tid 1: 0 ) v1 2024-02-06T21:41:38.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.093+0000 7fdf1a7fc640 10 client.4274.objecter _finish_command 1 = system:0 2024-02-06T21:41:38.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.093+0000 7fdf1a7fc640 20 client.4274.objecter put_session s=0x7fdf08072f90 osd=3 4 2024-02-06T21:41:38.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.093+0000 7fdf1a7fc640 15 client.4274.objecter _session_command_op_remove 3 1 2024-02-06T21:41:38.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.107+0000 7fdf2bc63640 10 client.4274.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-06T21:41:38.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.107+0000 7fdf2bc63640 15 client.4274.objecter _session_command_op_assign -1 2 2024-02-06T21:41:38.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.107+0000 7fdf2bc63640 20 client.4274.objecter _get_session s=0x7fdf08072f90 osd=3 3 2024-02-06T21:41:38.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.107+0000 7fdf2bc63640 20 client.4274.objecter put_session s=0x7fdf08072f90 osd=3 3 2024-02-06T21:41:38.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.107+0000 7fdf2bc63640 20 client.4274.objecter _get_session s=0x7fdf08072f90 osd=3 3 2024-02-06T21:41:38.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.107+0000 7fdf2bc63640 15 client.4274.objecter _session_command_op_remove -1 2 2024-02-06T21:41:38.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.107+0000 7fdf2bc63640 20 client.4274.objecter get_session s=0x7fdf08072f90 osd=3 3 2024-02-06T21:41:38.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.107+0000 7fdf2bc63640 15 client.4274.objecter _session_command_op_assign 3 2 2024-02-06T21:41:38.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.107+0000 7fdf2bc63640 20 client.4274.objecter put_session s=0x7fdf08072f90 osd=3 4 2024-02-06T21:41:38.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.107+0000 7fdf2bc63640 10 client.4274.objecter _send_command 2 2024-02-06T21:41:38.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.107+0000 7fdf2bc63640 1 -- 172.21.15.120:0/403538856 --> [v2:172.21.15.120:6800/3324566974,v1:172.21.15.120:6801/3324566974] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdf241d3550 con 0x7fdf08073660 2024-02-06T21:41:38.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.113+0000 7f2872ffd640 1 -- 172.21.15.120:0/214071414 <== osd.4 v2:172.21.15.135:6802/2191547918 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2864076100 con 0x7f2864073630 2024-02-06T21:41:38.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.113+0000 7f2872ffd640 10 client.4306.objecter ms_dispatch 0x7f2884000b90 command_reply(tid 1: 0 ) v1 2024-02-06T21:41:38.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.113+0000 7f2872ffd640 10 client.4306.objecter _finish_command 1 = system:0 2024-02-06T21:41:38.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.113+0000 7f2872ffd640 20 client.4306.objecter put_session s=0x7f2864072f60 osd=4 4 2024-02-06T21:41:38.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.113+0000 7f2872ffd640 15 client.4306.objecter _session_command_op_remove 4 1 2024-02-06T21:41:38.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.113+0000 7f5792ffd640 1 -- 172.21.15.120:0/3893330556 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f579c0b7400 con 0x7f57940cf6e0 2024-02-06T21:41:38.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.114+0000 7f5792ffd640 1 --2- 172.21.15.120:0/3893330556 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f578006d1f0 0x7f578006f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.114+0000 7f57a24d9640 1 --2- 172.21.15.120:0/3893330556 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f578006d1f0 0x7f578006f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.114+0000 7f57a24d9640 10 monclient: get_auth_request con 0x7f578006d1f0 auth_method 0 2024-02-06T21:41:38.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.115+0000 7f5792ffd640 10 client.4312.objecter ms_handle_connect 0x7f578006d1f0 2024-02-06T21:41:38.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.115+0000 7f57a24d9640 1 --2- 172.21.15.120:0/3893330556 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f578006d1f0 0x7f578006f6b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f578c006fd0 tx=0x7f578c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.116+0000 7f037a7fc640 1 -- 172.21.15.120:0/132820862 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0374032070 con 0x7f037c032110 2024-02-06T21:41:38.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.116+0000 7f037a7fc640 1 --2- 172.21.15.120:0/132820862 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f036806d220 0x7f036806f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.117+0000 7f0389fe1640 1 --2- 172.21.15.120:0/132820862 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f036806d220 0x7f036806f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.117+0000 7f0389fe1640 10 monclient: get_auth_request con 0x7f036806d220 auth_method 0 2024-02-06T21:41:38.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.117+0000 7f1523fff640 1 -- 172.21.15.120:0/3429897340 <== osd.1 v2:172.21.15.120:6808/3348315891 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1528076100 con 0x7f1528073630 2024-02-06T21:41:38.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.117+0000 7f1523fff640 10 client.4280.objecter ms_dispatch 0x7f153c000b90 command_reply(tid 1: 0 ) v1 2024-02-06T21:41:38.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.117+0000 7f1523fff640 10 client.4280.objecter _finish_command 1 = system:0 2024-02-06T21:41:38.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.117+0000 7f1523fff640 20 client.4280.objecter put_session s=0x7f1528072f60 osd=1 4 2024-02-06T21:41:38.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.117+0000 7f1523fff640 15 client.4280.objecter _session_command_op_remove 1 1 2024-02-06T21:41:38.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.117+0000 7f037a7fc640 10 client.4311.objecter ms_handle_connect 0x7f036806d220 2024-02-06T21:41:38.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.117+0000 7f0389fe1640 1 --2- 172.21.15.120:0/132820862 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f036806d220 0x7f036806f6e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0380009910 tx=0x7f0380008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 1 -- 172.21.15.120:0/3893330556 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f579c0f6190 con 0x7f57940cf6e0 2024-02-06T21:41:38.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 10 client.4312.objecter ms_dispatch 0x7f579c000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:38.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 3 client.4312.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:38.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 3 client.4312.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:38.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 10 client.4312.objecter checking command 1 2024-02-06T21:41:38.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 1 --2- 172.21.15.120:0/3893330556 >> [v2:172.21.15.120:6824/3229495265,v1:172.21.15.120:6825/3229495265] conn(0x7f5780073630 0x7f5780075a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 20 client.4312.objecter _get_session s=0x7f5780072f60 osd=2 3 2024-02-06T21:41:38.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 20 client.4312.objecter put_session s=0x7f5780072f60 osd=2 3 2024-02-06T21:41:38.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 15 client.4312.objecter _session_command_op_remove -1 1 2024-02-06T21:41:38.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 20 client.4312.objecter _get_session s=0x7f5780072f60 osd=2 3 2024-02-06T21:41:38.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 20 client.4312.objecter get_session s=0x7f5780072f60 osd=2 3 2024-02-06T21:41:38.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 15 client.4312.objecter _session_command_op_assign 2 1 2024-02-06T21:41:38.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 20 client.4312.objecter put_session s=0x7f5780072f60 osd=2 4 2024-02-06T21:41:38.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 10 client.4312.objecter _send_command 1 2024-02-06T21:41:38.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 1 -- 172.21.15.120:0/3893330556 --> [v2:172.21.15.120:6824/3229495265,v1:172.21.15.120:6825/3229495265] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5780076100 con 0x7f5780073630 2024-02-06T21:41:38.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 20 client.4312.objecter dump_active .. 0 homeless 2024-02-06T21:41:38.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 1 -- 172.21.15.120:0/3893330556 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f579c0f1d30 con 0x7f57940cf6e0 2024-02-06T21:41:38.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.119+0000 7f5792ffd640 10 monclient: handle_get_version_reply finishing 1 version 23 2024-02-06T21:41:38.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.120+0000 7f57a1cd8640 1 --2- 172.21.15.120:0/3893330556 >> [v2:172.21.15.120:6824/3229495265,v1:172.21.15.120:6825/3229495265] conn(0x7f5780073630 0x7f5780075a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.120+0000 7f57a1cd8640 10 monclient: get_auth_request con 0x7f5780073630 auth_method 0 2024-02-06T21:41:38.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.122+0000 7f037a7fc640 1 -- 172.21.15.120:0/132820862 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f037407a6b0 con 0x7f037c032110 2024-02-06T21:41:38.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.122+0000 7f037a7fc640 10 client.4311.objecter ms_dispatch 0x7f0384000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:38.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.122+0000 7f037a7fc640 3 client.4311.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:38.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.122+0000 7f037a7fc640 3 client.4311.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:38.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.122+0000 7f037a7fc640 10 client.4311.objecter checking command 1 2024-02-06T21:41:38.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.122+0000 7f037a7fc640 1 --2- 172.21.15.120:0/132820862 >> [v2:172.21.15.135:6818/563193066,v1:172.21.15.135:6819/563193066] conn(0x7f0368073660 0x7f0368075a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.174 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.122+0000 7f037a7fc640 20 client.4311.objecter _get_session s=0x7f0368072f90 osd=5 3 2024-02-06T21:41:38.174 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.122+0000 7f037a7fc640 20 client.4311.objecter put_session s=0x7f0368072f90 osd=5 3 2024-02-06T21:41:38.174 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.122+0000 7f037a7fc640 15 client.4311.objecter _session_command_op_remove -1 1 2024-02-06T21:41:38.174 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.122+0000 7f037a7fc640 20 client.4311.objecter _get_session s=0x7f0368072f90 osd=5 3 2024-02-06T21:41:38.174 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.122+0000 7f037a7fc640 20 client.4311.objecter get_session s=0x7f0368072f90 osd=5 3 2024-02-06T21:41:38.174 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.122+0000 7f037a7fc640 15 client.4311.objecter _session_command_op_assign 5 1 2024-02-06T21:41:38.174 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.122+0000 7f037a7fc640 20 client.4311.objecter put_session s=0x7f0368072f90 osd=5 4 2024-02-06T21:41:38.174 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.122+0000 7f037a7fc640 10 client.4311.objecter _send_command 1 2024-02-06T21:41:38.174 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.123+0000 7f037a7fc640 1 -- 172.21.15.120:0/132820862 --> [v2:172.21.15.135:6818/563193066,v1:172.21.15.135:6819/563193066] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0368076130 con 0x7f0368073660 2024-02-06T21:41:38.174 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.123+0000 7f037a7fc640 20 client.4311.objecter dump_active .. 0 homeless 2024-02-06T21:41:38.174 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.123+0000 7f038a7e2640 1 --2- 172.21.15.120:0/132820862 >> [v2:172.21.15.135:6818/563193066,v1:172.21.15.135:6819/563193066] conn(0x7f0368073660 0x7f0368075a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.174 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.124+0000 7f038a7e2640 10 monclient: get_auth_request con 0x7f0368073660 auth_method 0 2024-02-06T21:41:38.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.124+0000 7f57a1cd8640 1 --2- 172.21.15.120:0/3893330556 >> [v2:172.21.15.120:6824/3229495265,v1:172.21.15.120:6825/3229495265] conn(0x7f5780073630 0x7f5780075a50 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.124+0000 7f5792ffd640 10 client.4312.objecter ms_handle_connect 0x7f5780073630 2024-02-06T21:41:38.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.125+0000 7f037a7fc640 1 -- 172.21.15.120:0/132820862 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0374001d70 con 0x7f037c032110 2024-02-06T21:41:38.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.125+0000 7f037a7fc640 10 monclient: handle_get_version_reply finishing 1 version 23 2024-02-06T21:41:38.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.128+0000 7f037a7fc640 10 client.4311.objecter ms_handle_connect 0x7f0368073660 2024-02-06T21:41:38.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.128+0000 7f038a7e2640 1 --2- 172.21.15.120:0/132820862 >> [v2:172.21.15.135:6818/563193066,v1:172.21.15.135:6819/563193066] conn(0x7f0368073660 0x7f0368075a80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.176 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.132+0000 7fdf1a7fc640 1 -- 172.21.15.120:0/403538856 <== osd.3 v2:172.21.15.120:6800/3324566974 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdf241d3550 con 0x7fdf08073660 2024-02-06T21:41:38.176 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.132+0000 7fdf1a7fc640 10 client.4274.objecter ms_dispatch 0x7fdf24000b90 command_reply(tid 2: 0 ) v1 2024-02-06T21:41:38.176 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.132+0000 7fdf1a7fc640 10 client.4274.objecter _finish_command 2 = system:0 2024-02-06T21:41:38.176 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.132+0000 7fdf1a7fc640 20 client.4274.objecter put_session s=0x7fdf08072f90 osd=3 4 2024-02-06T21:41:38.176 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.132+0000 7fdf1a7fc640 15 client.4274.objecter _session_command_op_remove 3 2 2024-02-06T21:41:38.177 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 10 librados: watch_flush enter 2024-02-06T21:41:38.177 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 10 librados: watch_flush exit 2024-02-06T21:41:38.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 10 client.4274.objecter close_session for osd.3 2024-02-06T21:41:38.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 1 -- 172.21.15.120:0/403538856 >> [v2:172.21.15.120:6800/3324566974,v1:172.21.15.120:6801/3324566974] conn(0x7fdf08073660 msgr2=0x7fdf08075a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 1 --2- 172.21.15.120:0/403538856 >> [v2:172.21.15.120:6800/3324566974,v1:172.21.15.120:6801/3324566974] conn(0x7fdf08073660 0x7fdf08075a80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 20 client.4274.objecter put_session s=0x7fdf08072f90 osd=3 1 2024-02-06T21:41:38.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 20 client.4274.objecter shutdown clearing up homeless session... 2024-02-06T21:41:38.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 10 client.4274.objecter successfully canceled tick 2024-02-06T21:41:38.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 1 -- 172.21.15.120:0/403538856 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fdf0806d220 msgr2=0x7fdf0806f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 1 --2- 172.21.15.120:0/403538856 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fdf0806d220 0x7fdf0806f6e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdf14006fd0 tx=0x7fdf14008040 comp rx=0 tx=0).stop 2024-02-06T21:41:38.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 10 monclient: shutdown 2024-02-06T21:41:38.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 1 -- 172.21.15.120:0/403538856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf2413ba80 msgr2=0x7fdf241d3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 1 --2- 172.21.15.120:0/403538856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf2413ba80 0x7fdf241d3d70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c03a040 tx=0x7fdf1c004ab0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2a200640 1 -- 172.21.15.120:0/403538856 reap_dead start 2024-02-06T21:41:38.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 1 -- 172.21.15.120:0/403538856 shutdown_connections 2024-02-06T21:41:38.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 1 -- 172.21.15.120:0/403538856 >> 172.21.15.120:0/403538856 conn(0x7fdf240b07a0 msgr2=0x7fdf240b0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 1 -- 172.21.15.120:0/403538856 shutdown_connections 2024-02-06T21:41:38.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 1 -- 172.21.15.120:0/403538856 wait complete. 2024-02-06T21:41:38.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.133+0000 7fdf2bc63640 1 librados: shutdown 2024-02-06T21:41:38.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.140+0000 7f0b91ffb640 1 -- 172.21.15.120:0/734218230 <== osd.7 v2:172.21.15.135:6825/3740758337 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0b80076100 con 0x7f0b80073630 2024-02-06T21:41:38.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.140+0000 7f0b91ffb640 10 client.4302.objecter ms_dispatch 0x7f0b9c000b90 command_reply(tid 1: 0 ) v1 2024-02-06T21:41:38.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.140+0000 7f0b91ffb640 10 client.4302.objecter _finish_command 1 = system:0 2024-02-06T21:41:38.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.140+0000 7f0b91ffb640 20 client.4302.objecter put_session s=0x7f0b80072f60 osd=7 4 2024-02-06T21:41:38.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.140+0000 7f0b91ffb640 15 client.4302.objecter _session_command_op_remove 7 1 2024-02-06T21:41:38.180 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.142+0000 7f288359e640 10 client.4306.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-06T21:41:38.180 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.143+0000 7f288359e640 15 client.4306.objecter _session_command_op_assign -1 2 2024-02-06T21:41:38.180 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.143+0000 7f288359e640 20 client.4306.objecter _get_session s=0x7f2864072f60 osd=4 3 2024-02-06T21:41:38.180 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.143+0000 7f288359e640 20 client.4306.objecter put_session s=0x7f2864072f60 osd=4 3 2024-02-06T21:41:38.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.143+0000 7f288359e640 20 client.4306.objecter _get_session s=0x7f2864072f60 osd=4 3 2024-02-06T21:41:38.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.143+0000 7f288359e640 15 client.4306.objecter _session_command_op_remove -1 2 2024-02-06T21:41:38.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.143+0000 7f288359e640 20 client.4306.objecter get_session s=0x7f2864072f60 osd=4 3 2024-02-06T21:41:38.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.143+0000 7f288359e640 15 client.4306.objecter _session_command_op_assign 4 2 2024-02-06T21:41:38.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.143+0000 7f288359e640 20 client.4306.objecter put_session s=0x7f2864072f60 osd=4 4 2024-02-06T21:41:38.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.143+0000 7f288359e640 10 client.4306.objecter _send_command 2 2024-02-06T21:41:38.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.143+0000 7f288359e640 1 -- 172.21.15.120:0/214071414 --> [v2:172.21.15.135:6802/2191547918,v1:172.21.15.135:6803/2191547918] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f287c0302f0 con 0x7f2864073630 2024-02-06T21:41:38.182 INFO:teuthology.orchestra.run.smithi120.stdout:73014444040 2024-02-06T21:41:38.182 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-02-06T21:41:38.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.149+0000 7f57797fa640 1 -- 172.21.15.120:0/3775077712 <== osd.0 v2:172.21.15.120:6816/2062960776 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f57680760b0 con 0x7f57680735e0 2024-02-06T21:41:38.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.149+0000 7f57797fa640 10 client.4297.objecter ms_dispatch 0x7f577c000b90 command_reply(tid 1: 0 ) v1 2024-02-06T21:41:38.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.149+0000 7f57797fa640 10 client.4297.objecter _finish_command 1 = system:0 2024-02-06T21:41:38.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.149+0000 7f57797fa640 20 client.4297.objecter put_session s=0x7f5768072f10 osd=0 4 2024-02-06T21:41:38.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.149+0000 7f57797fa640 15 client.4297.objecter _session_command_op_remove 0 1 2024-02-06T21:41:38.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.150+0000 7f1540899640 10 client.4280.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-06T21:41:38.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.150+0000 7f1540899640 15 client.4280.objecter _session_command_op_assign -1 2 2024-02-06T21:41:38.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.150+0000 7f1540899640 20 client.4280.objecter _get_session s=0x7f1528072f60 osd=1 3 2024-02-06T21:41:38.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.150+0000 7f1540899640 20 client.4280.objecter put_session s=0x7f1528072f60 osd=1 3 2024-02-06T21:41:38.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.150+0000 7f1540899640 20 client.4280.objecter _get_session s=0x7f1528072f60 osd=1 3 2024-02-06T21:41:38.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.150+0000 7f1540899640 15 client.4280.objecter _session_command_op_remove -1 2 2024-02-06T21:41:38.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.150+0000 7f1540899640 20 client.4280.objecter get_session s=0x7f1528072f60 osd=1 3 2024-02-06T21:41:38.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.150+0000 7f1540899640 15 client.4280.objecter _session_command_op_assign 1 2 2024-02-06T21:41:38.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.150+0000 7f1540899640 20 client.4280.objecter put_session s=0x7f1528072f60 osd=1 4 2024-02-06T21:41:38.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.150+0000 7f1540899640 10 client.4280.objecter _send_command 2 2024-02-06T21:41:38.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.150+0000 7f1540899640 1 -- 172.21.15.120:0/3429897340 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f153c143260 con 0x7f1528073630 2024-02-06T21:41:38.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f0ba327b640 10 client.4302.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-06T21:41:38.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f0ba327b640 15 client.4302.objecter _session_command_op_assign -1 2 2024-02-06T21:41:38.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f0ba327b640 20 client.4302.objecter _get_session s=0x7f0b80072f60 osd=7 3 2024-02-06T21:41:38.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f0ba327b640 20 client.4302.objecter put_session s=0x7f0b80072f60 osd=7 3 2024-02-06T21:41:38.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f0ba327b640 20 client.4302.objecter _get_session s=0x7f0b80072f60 osd=7 3 2024-02-06T21:41:38.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f0ba327b640 15 client.4302.objecter _session_command_op_remove -1 2 2024-02-06T21:41:38.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f0ba327b640 20 client.4302.objecter get_session s=0x7f0b80072f60 osd=7 3 2024-02-06T21:41:38.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f0ba327b640 15 client.4302.objecter _session_command_op_assign 7 2 2024-02-06T21:41:38.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f0ba327b640 20 client.4302.objecter put_session s=0x7f0b80072f60 osd=7 4 2024-02-06T21:41:38.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f0ba327b640 10 client.4302.objecter _send_command 2 2024-02-06T21:41:38.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f0ba327b640 1 -- 172.21.15.120:0/734218230 --> [v2:172.21.15.135:6825/3740758337,v1:172.21.15.135:6827/3740758337] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0b9c0adc90 con 0x7f0b80073630 2024-02-06T21:41:38.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f1523fff640 1 -- 172.21.15.120:0/3429897340 <== osd.1 v2:172.21.15.120:6808/3348315891 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1530009280 con 0x7f1528073630 2024-02-06T21:41:38.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f1523fff640 10 client.4280.objecter ms_dispatch 0x7f153c000b90 command_reply(tid 2: 0 ) v1 2024-02-06T21:41:38.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f1523fff640 10 client.4280.objecter _finish_command 2 = system:0 2024-02-06T21:41:38.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f1523fff640 20 client.4280.objecter put_session s=0x7f1528072f60 osd=1 4 2024-02-06T21:41:38.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.156+0000 7f1523fff640 15 client.4280.objecter _session_command_op_remove 1 2 2024-02-06T21:41:38.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 10 librados: watch_flush enter 2024-02-06T21:41:38.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 10 librados: watch_flush exit 2024-02-06T21:41:38.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 10 client.4280.objecter close_session for osd.1 2024-02-06T21:41:38.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 1 -- 172.21.15.120:0/3429897340 >> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] conn(0x7f1528073630 msgr2=0x7f1528075a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 1 --2- 172.21.15.120:0/3429897340 >> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] conn(0x7f1528073630 0x7f1528075a50 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 20 client.4280.objecter put_session s=0x7f1528072f60 osd=1 1 2024-02-06T21:41:38.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 20 client.4280.objecter shutdown clearing up homeless session... 2024-02-06T21:41:38.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 10 client.4280.objecter successfully canceled tick 2024-02-06T21:41:38.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 1 -- 172.21.15.120:0/3429897340 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f152806d1f0 msgr2=0x7f152806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 1 --2- 172.21.15.120:0/3429897340 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f152806d1f0 0x7f152806f6b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f152c0015f0 tx=0x7f152c007a40 comp rx=0 tx=0).stop 2024-02-06T21:41:38.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 10 monclient: shutdown 2024-02-06T21:41:38.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 1 -- 172.21.15.120:0/3429897340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f153c13ff70 msgr2=0x7f153c139bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 1 --2- 172.21.15.120:0/3429897340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f153c13ff70 0x7f153c139bf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1534037040 tx=0x7f1534004a40 comp rx=0 tx=0).stop 2024-02-06T21:41:38.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f153bfff640 1 -- 172.21.15.120:0/3429897340 reap_dead start 2024-02-06T21:41:38.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 1 -- 172.21.15.120:0/3429897340 shutdown_connections 2024-02-06T21:41:38.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 1 -- 172.21.15.120:0/3429897340 >> 172.21.15.120:0/3429897340 conn(0x7f153c066df0 msgr2=0x7f153c135e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 1 -- 172.21.15.120:0/3429897340 shutdown_connections 2024-02-06T21:41:38.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 1 -- 172.21.15.120:0/3429897340 wait complete. 2024-02-06T21:41:38.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.157+0000 7f1540899640 1 librados: shutdown 2024-02-06T21:41:38.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.161+0000 7f2872ffd640 1 -- 172.21.15.120:0/214071414 <== osd.4 v2:172.21.15.135:6802/2191547918 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f287c0302f0 con 0x7f2864073630 2024-02-06T21:41:38.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.161+0000 7f2872ffd640 10 client.4306.objecter ms_dispatch 0x7f2884000b90 command_reply(tid 2: 0 ) v1 2024-02-06T21:41:38.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.161+0000 7f2872ffd640 10 client.4306.objecter _finish_command 2 = system:0 2024-02-06T21:41:38.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.161+0000 7f2872ffd640 20 client.4306.objecter put_session s=0x7f2864072f60 osd=4 4 2024-02-06T21:41:38.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.161+0000 7f2872ffd640 15 client.4306.objecter _session_command_op_remove 4 2 2024-02-06T21:41:38.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 10 librados: watch_flush enter 2024-02-06T21:41:38.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 10 librados: watch_flush exit 2024-02-06T21:41:38.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 10 client.4306.objecter close_session for osd.4 2024-02-06T21:41:38.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 1 -- 172.21.15.120:0/214071414 >> [v2:172.21.15.135:6802/2191547918,v1:172.21.15.135:6803/2191547918] conn(0x7f2864073630 msgr2=0x7f2864075a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 1 --2- 172.21.15.120:0/214071414 >> [v2:172.21.15.135:6802/2191547918,v1:172.21.15.135:6803/2191547918] conn(0x7f2864073630 0x7f2864075a50 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 20 client.4306.objecter put_session s=0x7f2864072f60 osd=4 1 2024-02-06T21:41:38.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 20 client.4306.objecter shutdown clearing up homeless session... 2024-02-06T21:41:38.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 10 client.4306.objecter successfully canceled tick 2024-02-06T21:41:38.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 1 -- 172.21.15.120:0/214071414 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f286406d1f0 msgr2=0x7f286406f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 1 --2- 172.21.15.120:0/214071414 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f286406d1f0 0x7f286406f6b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2884068c70 tx=0x7f288406e040 comp rx=0 tx=0).stop 2024-02-06T21:41:38.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 10 monclient: shutdown 2024-02-06T21:41:38.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 1 -- 172.21.15.120:0/214071414 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f287c02e1f0 msgr2=0x7f287c0caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 1 --2- 172.21.15.120:0/214071414 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f287c02e1f0 0x7f287c0caf40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2878000c00 tx=0x7f2878004730 comp rx=0 tx=0).stop 2024-02-06T21:41:38.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f2882d9d640 1 -- 172.21.15.120:0/214071414 reap_dead start 2024-02-06T21:41:38.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 1 -- 172.21.15.120:0/214071414 shutdown_connections 2024-02-06T21:41:38.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 1 -- 172.21.15.120:0/214071414 >> 172.21.15.120:0/214071414 conn(0x7f287c036610 msgr2=0x7f287c0cec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 1 -- 172.21.15.120:0/214071414 shutdown_connections 2024-02-06T21:41:38.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 1 -- 172.21.15.120:0/214071414 wait complete. 2024-02-06T21:41:38.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.162+0000 7f288359e640 1 librados: shutdown 2024-02-06T21:41:38.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.167+0000 7f578304a640 10 client.4297.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-06T21:41:38.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.167+0000 7f578304a640 15 client.4297.objecter _session_command_op_assign -1 2 2024-02-06T21:41:38.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.167+0000 7f578304a640 20 client.4297.objecter _get_session s=0x7f5768072f10 osd=0 3 2024-02-06T21:41:38.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.167+0000 7f578304a640 20 client.4297.objecter put_session s=0x7f5768072f10 osd=0 3 2024-02-06T21:41:38.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.167+0000 7f578304a640 20 client.4297.objecter _get_session s=0x7f5768072f10 osd=0 3 2024-02-06T21:41:38.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.167+0000 7f578304a640 15 client.4297.objecter _session_command_op_remove -1 2 2024-02-06T21:41:38.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.167+0000 7f578304a640 20 client.4297.objecter get_session s=0x7f5768072f10 osd=0 3 2024-02-06T21:41:38.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.167+0000 7f578304a640 15 client.4297.objecter _session_command_op_assign 0 2 2024-02-06T21:41:38.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.167+0000 7f578304a640 20 client.4297.objecter put_session s=0x7f5768072f10 osd=0 4 2024-02-06T21:41:38.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.167+0000 7f578304a640 10 client.4297.objecter _send_command 2 2024-02-06T21:41:38.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.167+0000 7f578304a640 1 -- 172.21.15.120:0/3775077712 --> [v2:172.21.15.120:6816/2062960776,v1:172.21.15.120:6817/2062960776] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f577c04ef90 con 0x7f57680735e0 2024-02-06T21:41:38.194 INFO:teuthology.orchestra.run.smithi120.stdout:77309411336 2024-02-06T21:41:38.195 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-02-06T21:41:38.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.173+0000 7f037a7fc640 1 -- 172.21.15.120:0/132820862 <== osd.5 v2:172.21.15.135:6818/563193066 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0368076130 con 0x7f0368073660 2024-02-06T21:41:38.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.173+0000 7f037a7fc640 10 client.4311.objecter ms_dispatch 0x7f0384000b90 command_reply(tid 1: 0 ) v1 2024-02-06T21:41:38.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.173+0000 7f037a7fc640 10 client.4311.objecter _finish_command 1 = system:0 2024-02-06T21:41:38.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.173+0000 7f037a7fc640 20 client.4311.objecter put_session s=0x7f0368072f90 osd=5 4 2024-02-06T21:41:38.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.173+0000 7f037a7fc640 15 client.4311.objecter _session_command_op_remove 5 1 2024-02-06T21:41:38.196 INFO:teuthology.orchestra.run.smithi120.stdout:73014444040 2024-02-06T21:41:38.196 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-02-06T21:41:38.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.176+0000 7f0b91ffb640 1 -- 172.21.15.120:0/734218230 <== osd.7 v2:172.21.15.135:6825/3740758337 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0b98009550 con 0x7f0b80073630 2024-02-06T21:41:38.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.176+0000 7f0b91ffb640 10 client.4302.objecter ms_dispatch 0x7f0b9c000b90 command_reply(tid 2: 0 ) v1 2024-02-06T21:41:38.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.176+0000 7f0b91ffb640 10 client.4302.objecter _finish_command 2 = system:0 2024-02-06T21:41:38.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.176+0000 7f0b91ffb640 20 client.4302.objecter put_session s=0x7f0b80072f60 osd=7 4 2024-02-06T21:41:38.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.176+0000 7f0b91ffb640 15 client.4302.objecter _session_command_op_remove 7 2 2024-02-06T21:41:38.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 10 librados: watch_flush enter 2024-02-06T21:41:38.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 10 librados: watch_flush exit 2024-02-06T21:41:38.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 10 client.4302.objecter close_session for osd.7 2024-02-06T21:41:38.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 1 -- 172.21.15.120:0/734218230 >> [v2:172.21.15.135:6825/3740758337,v1:172.21.15.135:6827/3740758337] conn(0x7f0b80073630 msgr2=0x7f0b80075a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 1 --2- 172.21.15.120:0/734218230 >> [v2:172.21.15.135:6825/3740758337,v1:172.21.15.135:6827/3740758337] conn(0x7f0b80073630 0x7f0b80075a50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 20 client.4302.objecter put_session s=0x7f0b80072f60 osd=7 1 2024-02-06T21:41:38.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 20 client.4302.objecter shutdown clearing up homeless session... 2024-02-06T21:41:38.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 10 client.4302.objecter successfully canceled tick 2024-02-06T21:41:38.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 1 -- 172.21.15.120:0/734218230 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f0b8006d1f0 msgr2=0x7f0b8006f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 1 --2- 172.21.15.120:0/734218230 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f0b8006d1f0 0x7f0b8006f6b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c006fd0 tx=0x7f0b8c008040 comp rx=0 tx=0).stop 2024-02-06T21:41:38.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 10 monclient: shutdown 2024-02-06T21:41:38.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 1 -- 172.21.15.120:0/734218230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b9c06b590 msgr2=0x7f0b9c06deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 1 --2- 172.21.15.120:0/734218230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b9c06b590 0x7f0b9c06deb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0b94009e50 tx=0x7f0b94004a40 comp rx=0 tx=0).stop 2024-02-06T21:41:38.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba1818640 1 -- 172.21.15.120:0/734218230 reap_dead start 2024-02-06T21:41:38.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 1 -- 172.21.15.120:0/734218230 shutdown_connections 2024-02-06T21:41:38.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 1 -- 172.21.15.120:0/734218230 >> 172.21.15.120:0/734218230 conn(0x7f0b9c0b0ae0 msgr2=0x7f0b9c0b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 1 -- 172.21.15.120:0/734218230 shutdown_connections 2024-02-06T21:41:38.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 1 -- 172.21.15.120:0/734218230 wait complete. 2024-02-06T21:41:38.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.177+0000 7f0ba327b640 1 librados: shutdown 2024-02-06T21:41:38.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f57797fa640 1 -- 172.21.15.120:0/3775077712 <== osd.0 v2:172.21.15.120:6816/2062960776 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f577c04ef90 con 0x7f57680735e0 2024-02-06T21:41:38.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f57797fa640 10 client.4297.objecter ms_dispatch 0x7f577c000b90 command_reply(tid 2: 0 ) v1 2024-02-06T21:41:38.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f57797fa640 10 client.4297.objecter _finish_command 2 = system:0 2024-02-06T21:41:38.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f57797fa640 20 client.4297.objecter put_session s=0x7f5768072f10 osd=0 4 2024-02-06T21:41:38.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f57797fa640 15 client.4297.objecter _session_command_op_remove 0 2 2024-02-06T21:41:38.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 10 librados: watch_flush enter 2024-02-06T21:41:38.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 10 librados: watch_flush exit 2024-02-06T21:41:38.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 10 client.4297.objecter close_session for osd.0 2024-02-06T21:41:38.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 1 -- 172.21.15.120:0/3775077712 >> [v2:172.21.15.120:6816/2062960776,v1:172.21.15.120:6817/2062960776] conn(0x7f57680735e0 msgr2=0x7f5768075a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 1 --2- 172.21.15.120:0/3775077712 >> [v2:172.21.15.120:6816/2062960776,v1:172.21.15.120:6817/2062960776] conn(0x7f57680735e0 0x7f5768075a00 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 20 client.4297.objecter put_session s=0x7f5768072f10 osd=0 1 2024-02-06T21:41:38.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 20 client.4297.objecter shutdown clearing up homeless session... 2024-02-06T21:41:38.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 10 client.4297.objecter successfully canceled tick 2024-02-06T21:41:38.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 1 -- 172.21.15.120:0/3775077712 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f576806d1a0 msgr2=0x7f576806f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 1 --2- 172.21.15.120:0/3775077712 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f576806d1a0 0x7f576806f660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f576c036000 tx=0x7f576c03d040 comp rx=0 tx=0).stop 2024-02-06T21:41:38.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 10 monclient: shutdown 2024-02-06T21:41:38.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 1 -- 172.21.15.120:0/3775077712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f577c06e780 msgr2=0x7f577c140330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 1 --2- 172.21.15.120:0/3775077712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f577c06e780 0x7f577c140330 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f577000c430 tx=0x7f577000c8f0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f57815e7640 1 -- 172.21.15.120:0/3775077712 reap_dead start 2024-02-06T21:41:38.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 1 -- 172.21.15.120:0/3775077712 shutdown_connections 2024-02-06T21:41:38.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f578304a640 1 -- 172.21.15.120:0/3775077712 >> 172.21.15.120:0/3775077712 conn(0x7f577c0b07a0 msgr2=0x7f577c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.182+0000 7f5792ffd640 1 -- 172.21.15.120:0/3893330556 <== osd.2 v2:172.21.15.120:6824/3229495265 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5780076100 con 0x7f5780073630 2024-02-06T21:41:38.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.183+0000 7f5792ffd640 10 client.4312.objecter ms_dispatch 0x7f579c000b90 command_reply(tid 1: 0 ) v1 2024-02-06T21:41:38.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.183+0000 7f5792ffd640 10 client.4312.objecter _finish_command 1 = system:0 2024-02-06T21:41:38.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.183+0000 7f5792ffd640 20 client.4312.objecter put_session s=0x7f5780072f60 osd=2 4 2024-02-06T21:41:38.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.183+0000 7f5792ffd640 15 client.4312.objecter _session_command_op_remove 2 1 2024-02-06T21:41:38.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.184+0000 7f578304a640 1 -- 172.21.15.120:0/3775077712 shutdown_connections 2024-02-06T21:41:38.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.184+0000 7f578304a640 1 -- 172.21.15.120:0/3775077712 wait complete. 2024-02-06T21:41:38.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.184+0000 7f578304a640 1 librados: shutdown 2024-02-06T21:41:38.205 INFO:teuthology.orchestra.run.smithi120.stdout:55834574854 2024-02-06T21:41:38.206 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-02-06T21:41:38.207 INFO:teuthology.orchestra.run.smithi120.stdout:55834574853 2024-02-06T21:41:38.207 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-02-06T21:41:38.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.196+0000 7f038c245640 10 client.4311.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-06T21:41:38.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.196+0000 7f038c245640 15 client.4311.objecter _session_command_op_assign -1 2 2024-02-06T21:41:38.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.196+0000 7f038c245640 20 client.4311.objecter _get_session s=0x7f0368072f90 osd=5 3 2024-02-06T21:41:38.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.196+0000 7f038c245640 20 client.4311.objecter put_session s=0x7f0368072f90 osd=5 3 2024-02-06T21:41:38.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.196+0000 7f038c245640 20 client.4311.objecter _get_session s=0x7f0368072f90 osd=5 3 2024-02-06T21:41:38.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.196+0000 7f038c245640 15 client.4311.objecter _session_command_op_remove -1 2 2024-02-06T21:41:38.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.196+0000 7f038c245640 20 client.4311.objecter get_session s=0x7f0368072f90 osd=5 3 2024-02-06T21:41:38.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.196+0000 7f038c245640 15 client.4311.objecter _session_command_op_assign 5 2 2024-02-06T21:41:38.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.196+0000 7f038c245640 20 client.4311.objecter put_session s=0x7f0368072f90 osd=5 4 2024-02-06T21:41:38.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.196+0000 7f038c245640 10 client.4311.objecter _send_command 2 2024-02-06T21:41:38.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.196+0000 7f038c245640 1 -- 172.21.15.120:0/132820862 --> [v2:172.21.15.135:6818/563193066,v1:172.21.15.135:6819/563193066] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f036c001240 con 0x7f0368073660 2024-02-06T21:41:38.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f57a473d640 10 client.4312.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-06T21:41:38.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f57a473d640 15 client.4312.objecter _session_command_op_assign -1 2 2024-02-06T21:41:38.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f57a473d640 20 client.4312.objecter _get_session s=0x7f5780072f60 osd=2 3 2024-02-06T21:41:38.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f57a473d640 20 client.4312.objecter put_session s=0x7f5780072f60 osd=2 3 2024-02-06T21:41:38.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f57a473d640 20 client.4312.objecter _get_session s=0x7f5780072f60 osd=2 3 2024-02-06T21:41:38.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f57a473d640 15 client.4312.objecter _session_command_op_remove -1 2 2024-02-06T21:41:38.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f57a473d640 20 client.4312.objecter get_session s=0x7f5780072f60 osd=2 3 2024-02-06T21:41:38.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f57a473d640 15 client.4312.objecter _session_command_op_assign 2 2 2024-02-06T21:41:38.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f57a473d640 20 client.4312.objecter put_session s=0x7f5780072f60 osd=2 4 2024-02-06T21:41:38.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f57a473d640 10 client.4312.objecter _send_command 2 2024-02-06T21:41:38.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f57a473d640 1 -- 172.21.15.120:0/3893330556 --> [v2:172.21.15.120:6824/3229495265,v1:172.21.15.120:6825/3229495265] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f57940ca990 con 0x7f5780073630 2024-02-06T21:41:38.211 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f037a7fc640 1 -- 172.21.15.120:0/132820862 <== osd.5 v2:172.21.15.135:6818/563193066 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f036c001240 con 0x7f0368073660 2024-02-06T21:41:38.211 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f037a7fc640 10 client.4311.objecter ms_dispatch 0x7f0384000b90 command_reply(tid 2: 0 ) v1 2024-02-06T21:41:38.211 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f037a7fc640 10 client.4311.objecter _finish_command 2 = system:0 2024-02-06T21:41:38.211 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f037a7fc640 20 client.4311.objecter put_session s=0x7f0368072f90 osd=5 4 2024-02-06T21:41:38.211 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f037a7fc640 15 client.4311.objecter _session_command_op_remove 5 2 2024-02-06T21:41:38.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f038c245640 10 librados: watch_flush enter 2024-02-06T21:41:38.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f038c245640 10 librados: watch_flush exit 2024-02-06T21:41:38.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.204+0000 7f038c245640 10 client.4311.objecter close_session for osd.5 2024-02-06T21:41:38.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 1 -- 172.21.15.120:0/132820862 >> [v2:172.21.15.135:6818/563193066,v1:172.21.15.135:6819/563193066] conn(0x7f0368073660 msgr2=0x7f0368075a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 1 --2- 172.21.15.120:0/132820862 >> [v2:172.21.15.135:6818/563193066,v1:172.21.15.135:6819/563193066] conn(0x7f0368073660 0x7f0368075a80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 20 client.4311.objecter put_session s=0x7f0368072f90 osd=5 1 2024-02-06T21:41:38.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 20 client.4311.objecter shutdown clearing up homeless session... 2024-02-06T21:41:38.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 10 client.4311.objecter successfully canceled tick 2024-02-06T21:41:38.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 1 -- 172.21.15.120:0/132820862 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f036806d220 msgr2=0x7f036806f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 1 --2- 172.21.15.120:0/132820862 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f036806d220 0x7f036806f6e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0380009910 tx=0x7f0380008040 comp rx=0 tx=0).stop 2024-02-06T21:41:38.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 10 monclient: shutdown 2024-02-06T21:41:38.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 1 -- 172.21.15.120:0/132820862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f037c032110 msgr2=0x7f037c0a3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 1 --2- 172.21.15.120:0/132820862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f037c032110 0x7f037c0a3320 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0374007f90 tx=0x7f0374001930 comp rx=0 tx=0).stop 2024-02-06T21:41:38.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038a7e2640 1 -- 172.21.15.120:0/132820862 reap_dead start 2024-02-06T21:41:38.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 1 -- 172.21.15.120:0/132820862 shutdown_connections 2024-02-06T21:41:38.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 1 -- 172.21.15.120:0/132820862 >> 172.21.15.120:0/132820862 conn(0x7f037c006250 msgr2=0x7f037c0c71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 1 -- 172.21.15.120:0/132820862 shutdown_connections 2024-02-06T21:41:38.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 1 -- 172.21.15.120:0/132820862 wait complete. 2024-02-06T21:41:38.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.205+0000 7f038c245640 1 librados: shutdown 2024-02-06T21:41:38.217 INFO:teuthology.orchestra.run.smithi120.stdout:73014444042 2024-02-06T21:41:38.217 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-02-06T21:41:38.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.216+0000 7f5792ffd640 1 -- 172.21.15.120:0/3893330556 <== osd.2 v2:172.21.15.120:6824/3229495265 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f57940ca990 con 0x7f5780073630 2024-02-06T21:41:38.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.216+0000 7f5792ffd640 10 client.4312.objecter ms_dispatch 0x7f579c000b90 command_reply(tid 2: 0 ) v1 2024-02-06T21:41:38.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.216+0000 7f5792ffd640 10 client.4312.objecter _finish_command 2 = system:0 2024-02-06T21:41:38.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.216+0000 7f5792ffd640 20 client.4312.objecter put_session s=0x7f5780072f60 osd=2 4 2024-02-06T21:41:38.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.216+0000 7f5792ffd640 15 client.4312.objecter _session_command_op_remove 2 2 2024-02-06T21:41:38.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 10 librados: watch_flush enter 2024-02-06T21:41:38.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 10 librados: watch_flush exit 2024-02-06T21:41:38.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 10 client.4312.objecter close_session for osd.2 2024-02-06T21:41:38.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 1 -- 172.21.15.120:0/3893330556 >> [v2:172.21.15.120:6824/3229495265,v1:172.21.15.120:6825/3229495265] conn(0x7f5780073630 msgr2=0x7f5780075a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 1 --2- 172.21.15.120:0/3893330556 >> [v2:172.21.15.120:6824/3229495265,v1:172.21.15.120:6825/3229495265] conn(0x7f5780073630 0x7f5780075a50 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 20 client.4312.objecter put_session s=0x7f5780072f60 osd=2 1 2024-02-06T21:41:38.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 20 client.4312.objecter shutdown clearing up homeless session... 2024-02-06T21:41:38.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 10 client.4312.objecter successfully canceled tick 2024-02-06T21:41:38.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 1 -- 172.21.15.120:0/3893330556 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f578006d1f0 msgr2=0x7f578006f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 1 --2- 172.21.15.120:0/3893330556 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f578006d1f0 0x7f578006f6b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f578c006fd0 tx=0x7f578c008040 comp rx=0 tx=0).stop 2024-02-06T21:41:38.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 10 monclient: shutdown 2024-02-06T21:41:38.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 1 -- 172.21.15.120:0/3893330556 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57940cf6e0 msgr2=0x7f57940cb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 1 --2- 172.21.15.120:0/3893330556 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57940cf6e0 0x7f57940cb870 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f579c02d340 tx=0x7f579c068b50 comp rx=0 tx=0).stop 2024-02-06T21:41:38.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a2cda640 1 -- 172.21.15.120:0/3893330556 reap_dead start 2024-02-06T21:41:38.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 1 -- 172.21.15.120:0/3893330556 shutdown_connections 2024-02-06T21:41:38.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 1 -- 172.21.15.120:0/3893330556 >> 172.21.15.120:0/3893330556 conn(0x7f57940061f0 msgr2=0x7f57940c3570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 1 -- 172.21.15.120:0/3893330556 shutdown_connections 2024-02-06T21:41:38.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.217+0000 7f57a473d640 1 -- 172.21.15.120:0/3893330556 wait complete. 2024-02-06T21:41:38.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.218+0000 7f57a473d640 1 librados: shutdown 2024-02-06T21:41:38.236 INFO:teuthology.orchestra.run.smithi120.stdout:73014444040 2024-02-06T21:41:38.236 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-02-06T21:41:38.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.283+0000 7fd6e0ff9640 1 -- 172.21.15.120:0/2285512625 <== osd.6 v2:172.21.15.135:6810/1080459988 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd6c8076100 con 0x7fd6c8073630 2024-02-06T21:41:38.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.283+0000 7fd6e0ff9640 10 client.4299.objecter ms_dispatch 0x7fd6e4000b90 command_reply(tid 1: 0 ) v1 2024-02-06T21:41:38.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.283+0000 7fd6e0ff9640 10 client.4299.objecter _finish_command 1 = system:0 2024-02-06T21:41:38.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.283+0000 7fd6e0ff9640 20 client.4299.objecter put_session s=0x7fd6c8072f60 osd=6 4 2024-02-06T21:41:38.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.283+0000 7fd6e0ff9640 15 client.4299.objecter _session_command_op_remove 6 1 2024-02-06T21:41:38.319 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.317+0000 7fd6e91bc640 10 client.4299.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-06T21:41:38.319 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.317+0000 7fd6e91bc640 15 client.4299.objecter _session_command_op_assign -1 2 2024-02-06T21:41:38.319 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.317+0000 7fd6e91bc640 20 client.4299.objecter _get_session s=0x7fd6c8072f60 osd=6 3 2024-02-06T21:41:38.320 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.317+0000 7fd6e91bc640 20 client.4299.objecter put_session s=0x7fd6c8072f60 osd=6 3 2024-02-06T21:41:38.320 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.318+0000 7fd6e91bc640 20 client.4299.objecter _get_session s=0x7fd6c8072f60 osd=6 3 2024-02-06T21:41:38.320 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.318+0000 7fd6e91bc640 15 client.4299.objecter _session_command_op_remove -1 2 2024-02-06T21:41:38.320 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.318+0000 7fd6e91bc640 20 client.4299.objecter get_session s=0x7fd6c8072f60 osd=6 3 2024-02-06T21:41:38.320 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.318+0000 7fd6e91bc640 15 client.4299.objecter _session_command_op_assign 6 2 2024-02-06T21:41:38.320 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.318+0000 7fd6e91bc640 20 client.4299.objecter put_session s=0x7fd6c8072f60 osd=6 4 2024-02-06T21:41:38.320 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.318+0000 7fd6e91bc640 10 client.4299.objecter _send_command 2 2024-02-06T21:41:38.320 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.318+0000 7fd6e91bc640 1 -- 172.21.15.120:0/2285512625 --> [v2:172.21.15.135:6810/1080459988,v1:172.21.15.135:6811/1080459988] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd6dc037330 con 0x7fd6c8073630 2024-02-06T21:41:38.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6e0ff9640 1 -- 172.21.15.120:0/2285512625 <== osd.6 v2:172.21.15.135:6810/1080459988 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd6d0001190 con 0x7fd6c8073630 2024-02-06T21:41:38.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6e0ff9640 10 client.4299.objecter ms_dispatch 0x7fd6e4000b90 command_reply(tid 2: 0 ) v1 2024-02-06T21:41:38.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6e0ff9640 10 client.4299.objecter _finish_command 2 = system:0 2024-02-06T21:41:38.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6e0ff9640 20 client.4299.objecter put_session s=0x7fd6c8072f60 osd=6 4 2024-02-06T21:41:38.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6e0ff9640 15 client.4299.objecter _session_command_op_remove 6 2 2024-02-06T21:41:38.331 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 10 librados: watch_flush enter 2024-02-06T21:41:38.331 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 10 librados: watch_flush exit 2024-02-06T21:41:38.331 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 10 client.4299.objecter close_session for osd.6 2024-02-06T21:41:38.331 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 1 -- 172.21.15.120:0/2285512625 >> [v2:172.21.15.135:6810/1080459988,v1:172.21.15.135:6811/1080459988] conn(0x7fd6c8073630 msgr2=0x7fd6c8075a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.331 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 1 --2- 172.21.15.120:0/2285512625 >> [v2:172.21.15.135:6810/1080459988,v1:172.21.15.135:6811/1080459988] conn(0x7fd6c8073630 0x7fd6c8075a50 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.331 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 20 client.4299.objecter put_session s=0x7fd6c8072f60 osd=6 1 2024-02-06T21:41:38.331 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 20 client.4299.objecter shutdown clearing up homeless session... 2024-02-06T21:41:38.331 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 10 client.4299.objecter successfully canceled tick 2024-02-06T21:41:38.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 1 -- 172.21.15.120:0/2285512625 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd6c806d1f0 msgr2=0x7fd6c806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 1 --2- 172.21.15.120:0/2285512625 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd6c806d1f0 0x7fd6c806f6b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8034000 tx=0x7fd6d8008030 comp rx=0 tx=0).stop 2024-02-06T21:41:38.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 10 monclient: shutdown 2024-02-06T21:41:38.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 1 -- 172.21.15.120:0/2285512625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6dc02eae0 msgr2=0x7fd6dc02ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 1 --2- 172.21.15.120:0/2285512625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6dc02eae0 0x7fd6dc02ef40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd6e404fd40 tx=0x7fd6e4069f50 comp rx=0 tx=0).stop 2024-02-06T21:41:38.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6e89bb640 1 -- 172.21.15.120:0/2285512625 reap_dead start 2024-02-06T21:41:38.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 1 -- 172.21.15.120:0/2285512625 shutdown_connections 2024-02-06T21:41:38.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.329+0000 7fd6ce7fc640 1 -- 172.21.15.120:0/2285512625 >> 172.21.15.120:0/2285512625 conn(0x7fd6dc0061a0 msgr2=0x7fd6dc0cb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.333 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.331+0000 7fd6ce7fc640 1 -- 172.21.15.120:0/2285512625 shutdown_connections 2024-02-06T21:41:38.333 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.332+0000 7fd6ce7fc640 1 -- 172.21.15.120:0/2285512625 wait complete. 2024-02-06T21:41:38.334 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.332+0000 7fd6ce7fc640 1 librados: shutdown 2024-02-06T21:41:38.358 INFO:teuthology.orchestra.run.smithi120.stdout:73014444040 2024-02-06T21:41:38.358 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-02-06T21:41:38.377 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.373+0000 7f763b9c8640 10 monclient: get_monmap_and_config 2024-02-06T21:41:38.377 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.373+0000 7f763b9c8640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.377 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.373+0000 7f763b9c8640 10 monclient: monmap: 2024-02-06T21:41:38.377 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.377 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.377 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:38.374539+0000 2024-02-06T21:41:38.377 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:38.374539+0000 2024-02-06T21:41:38.377 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.377 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.377 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:38.377 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:38.377 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:38.377 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:38.378 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:38.378 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.374+0000 7f763b9c8640 1 Processor -- start 2024-02-06T21:41:38.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.374+0000 7f763b9c8640 1 -- start start 2024-02-06T21:41:38.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.374+0000 7f763b9c8640 10 monclient: init 2024-02-06T21:41:38.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.374+0000 7f3ef67e6640 10 monclient: get_monmap_and_config 2024-02-06T21:41:38.379 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.374+0000 7f3ef67e6640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.379 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.374+0000 7f3ef67e6640 10 monclient: monmap: 2024-02-06T21:41:38.379 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.379 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.379 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:38.376138+0000 2024-02-06T21:41:38.379 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:38.376138+0000 2024-02-06T21:41:38.379 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.379 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.379 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:38.379 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:38.379 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:38.379 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:38.380 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:38.380 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.375+0000 7f763b9c8640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.375+0000 7f763b9c8640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-02-06T21:41:38.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.375+0000 7f763b9c8640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f763408b2a0 0x7f763414f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.375+0000 7f763b9c8640 10 monclient(hunting): picked mon.noname-c con 0x7f763408b2a0 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:38.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.375+0000 7f763b9c8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7634070cd0 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:38.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.375+0000 7f763b9c8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7634157880 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:38.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.375+0000 7f763b9c8640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7634070640 con 0x7f7634157880 2024-02-06T21:41:38.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.375+0000 7f763b9c8640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76340708b0 con 0x7f7634070cd0 2024-02-06T21:41:38.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.376+0000 7f3ef67e6640 1 Processor -- start 2024-02-06T21:41:38.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.376+0000 7f3ef67e6640 1 -- start start 2024-02-06T21:41:38.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.376+0000 7f3ef67e6640 10 monclient: init 2024-02-06T21:41:38.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.373+0000 7fa1d83d6640 10 monclient: get_monmap_and_config 2024-02-06T21:41:38.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.373+0000 7fa1d83d6640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.373+0000 7fa1d83d6640 10 monclient: monmap: 2024-02-06T21:41:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:38.374784+0000 2024-02-06T21:41:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:38.374784+0000 2024-02-06T21:41:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:38.383 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.376+0000 7fa1d83d6640 1 Processor -- start 2024-02-06T21:41:38.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-02-06T21:41:38.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3ef006e7c0 0x7f3ef006eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 10 monclient(hunting): picked mon.noname-a con 0x7f3ef006e7c0 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:38.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ef006f920 0x7f3ef0143d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.376+0000 7fa1d83d6640 1 -- start start 2024-02-06T21:41:38.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.376+0000 7fa1d83d6640 10 monclient: init 2024-02-06T21:41:38.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 10 monclient(hunting): picked mon.noname-b con 0x7f3ef006f920 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3ef0071110 0x7f3ef0144260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 10 monclient(hunting): picked mon.noname-c con 0x7f3ef0071110 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:38.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7fa1d83d6640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ef0146b30 con 0x7f3ef006e7c0 2024-02-06T21:41:38.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ef0072cd0 con 0x7f3ef0071110 2024-02-06T21:41:38.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef0072e40 con 0x7f3ef006f920 2024-02-06T21:41:38.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f3ef67e6640 10 monclient(hunting): authenticate will time out at 1328.715362s 2024-02-06T21:41:38.388 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7fa1d83d6640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-02-06T21:41:38.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7fa1d83d6640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa1d00b12c0 0x7fa1d013ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7fa1d83d6640 10 monclient(hunting): picked mon.noname-a con 0x7fa1d00b12c0 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:38.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7fa1d83d6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1d013bfc0 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:38.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7fa1d83d6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa1d0071900 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:38.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7fa1d83d6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d013f580 con 0x7fa1d013bfc0 2024-02-06T21:41:38.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7fa1d83d6640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d006ff70 con 0x7fa1d0071900 2024-02-06T21:41:38.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7fa1d83d6640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7fa1d83d6640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1d006fac0 con 0x7fa1d00b12c0 2024-02-06T21:41:38.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7fa1d83d6640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7fa1d83d6640 10 monclient(hunting): authenticate will time out at 1328.715518s 2024-02-06T21:41:38.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.375+0000 7f763b9c8640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.375+0000 7f763b9c8640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f763406f710 con 0x7f763408b2a0 2024-02-06T21:41:38.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.375+0000 7f763b9c8640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.375+0000 7f763b9c8640 10 monclient(hunting): authenticate will time out at 1328.713026s 2024-02-06T21:41:38.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.376+0000 7f7639764640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f763408b2a0 0x7f763414f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f7639764640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f763408b2a0 0x7f763414f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38628/0 (socket says 172.21.15.120:38628) 2024-02-06T21:41:38.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f7639764640 1 -- 172.21.15.120:0/64401410 learned_addr learned my addr 172.21.15.120:0/64401410 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f7639764640 10 monclient(hunting): get_auth_request con 0x7f763408b2a0 auth_method 0 2024-02-06T21:41:38.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f7639764640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f7639764640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.391 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.377+0000 7f7639764640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.391 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.378+0000 7f3eef7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ef006f920 0x7f3ef0143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.391 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.379+0000 7f3eef7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ef006f920 0x7f3ef0143d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:56940/0 (socket says 172.21.15.120:56940) 2024-02-06T21:41:38.391 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.379+0000 7f3eef7fe640 1 -- 172.21.15.120:0/3827070521 learned_addr learned my addr 172.21.15.120:0/3827070521 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.391 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.379+0000 7f3eef7fe640 10 monclient(hunting): get_auth_request con 0x7f3ef006f920 auth_method 0 2024-02-06T21:41:38.391 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.379+0000 7f3eef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.379+0000 7f3eef7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.379+0000 7f3eef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.380+0000 7f3eef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.380+0000 7f3eef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.380+0000 7f3eef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.381+0000 7f3eeffff640 1 --2- 172.21.15.120:0/3827070521 >> v2:172.21.15.120:3300/0 conn(0x7f3ef006e7c0 0x7f3ef006eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.381+0000 7f3eeffff640 10 monclient(hunting): get_auth_request con 0x7f3ef006e7c0 auth_method 0 2024-02-06T21:41:38.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.381+0000 7f3eeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.381+0000 7f3eeffff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.381+0000 7f3eeffff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.381+0000 7f3ef4d83640 1 --2- 172.21.15.120:0/3827070521 >> v2:172.21.15.135:3300/0 conn(0x7f3ef0071110 0x7f3ef0144260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.382+0000 7f3ef4d83640 10 monclient(hunting): get_auth_request con 0x7f3ef0071110 auth_method 0 2024-02-06T21:41:38.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.382+0000 7f3ef4d83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.382+0000 7f3ef4d83640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.382+0000 7f3ef4d83640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f3ef4d83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f3ef4d83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f3ef4d83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.381+0000 7fa1d6973640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fa1d0071900 0x7fa1d0071d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.120:58856/0 (socket says 172.21.15.120:58856) 2024-02-06T21:41:38.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.381+0000 7fa1d6973640 1 -- 172.21.15.120:0/3834191863 learned_addr learned my addr 172.21.15.120:0/3834191863 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:41:38.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.381+0000 7fa1d6172640 1 --2- 172.21.15.120:0/3834191863 >> v2:172.21.15.120:3300/0 conn(0x7fa1d00b12c0 0x7fa1d013ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.381+0000 7fa1d6172640 10 monclient(hunting): get_auth_request con 0x7fa1d00b12c0 auth_method 0 2024-02-06T21:41:38.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.381+0000 7fa1d6172640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.381+0000 7fa1d6172640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.381+0000 7fa1d6172640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f7639764640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f7639764640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f7639764640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f3eef7fe640 10 monclient(hunting): handle_auth_done global_id 4286 payload 274 2024-02-06T21:41:38.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f3eef7fe640 1 -- 172.21.15.120:0/3827070521 >> v2:172.21.15.135:3300/0 conn(0x7f3ef0071110 msgr2=0x7f3ef0144260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:38.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f3eef7fe640 1 --2- 172.21.15.120:0/3827070521 >> v2:172.21.15.135:3300/0 conn(0x7f3ef0071110 0x7f3ef0144260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.384+0000 7fa1d6172640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.384+0000 7fa1d6172640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.384+0000 7fa1d6172640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f3eef7fe640 1 -- 172.21.15.120:0/3827070521 >> v2:172.21.15.120:3300/0 conn(0x7f3ef006e7c0 msgr2=0x7f3ef006eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:38.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.385+0000 7f1f9e40f640 10 monclient: get_monmap_and_config 2024-02-06T21:41:38.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.385+0000 7f1f9e40f640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.385+0000 7f1f9e40f640 10 monclient: monmap: 2024-02-06T21:41:38.397 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.397 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.397 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:38.386535+0000 2024-02-06T21:41:38.397 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:38.386535+0000 2024-02-06T21:41:38.397 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.398 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.398 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:38.398 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:38.398 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:38.398 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:38.398 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:38.398 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f3eef7fe640 1 --2- 172.21.15.120:0/3827070521 >> v2:172.21.15.120:3300/0 conn(0x7f3ef006e7c0 0x7f3ef006eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f3eef7fe640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f3eef7fe640 1 monclient: found mon.noname-b 2024-02-06T21:41:38.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f3eef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f3eef7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.383+0000 7f3eef7fe640 1 -- 172.21.15.120:0/3827070521 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef0146fc0 con 0x7f3ef006f920 2024-02-06T21:41:38.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.386+0000 7f1f9e40f640 1 Processor -- start 2024-02-06T21:41:38.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.386+0000 7f1f9e40f640 1 -- start start 2024-02-06T21:41:38.401 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.387+0000 7f3eef7fe640 1 --2- 172.21.15.120:0/3827070521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ef006f920 0x7f3ef0143d20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0072730 tx=0x7f3ee002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb162a050bec1540 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.401 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.386+0000 7f1f9e40f640 10 monclient: init 2024-02-06T21:41:38.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.387+0000 7f1f9e40f640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.387+0000 7f1f9e40f640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-02-06T21:41:38.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.387+0000 7f1f9e40f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1f9806e7c0 0x7f1f9806eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.387+0000 7f1f9e40f640 10 monclient(hunting): picked mon.noname-a con 0x7f1f9806e7c0 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:38.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.387+0000 7f1f9e40f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f9806f920 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:38.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.387+0000 7f1f9e40f640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1f98071110 0x7f1f98146fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.387+0000 7f1f9e40f640 10 monclient(hunting): picked mon.noname-c con 0x7f1f98071110 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:38.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.387+0000 7f1f9e40f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f98146b30 con 0x7f1f9806f920 2024-02-06T21:41:38.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.387+0000 7f1f9e40f640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.387+0000 7f1f9e40f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f98072cd0 con 0x7f1f9806e7c0 2024-02-06T21:41:38.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.387+0000 7f1f9e40f640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.387+0000 7f1f9e40f640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f98072e10 con 0x7f1f98071110 2024-02-06T21:41:38.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.387+0000 7f1f9e40f640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.387+0000 7f1f9e40f640 10 monclient(hunting): authenticate will time out at 1328.725221s 2024-02-06T21:41:38.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.388+0000 7fa1d6172640 10 monclient(hunting): handle_auth_done global_id 4314 payload 274 2024-02-06T21:41:38.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.388+0000 7fa1d6172640 1 -- 172.21.15.120:0/3834191863 >> v1:172.21.15.135:6789/0 conn(0x7fa1d0071900 legacy=0x7fa1d0071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.388+0000 7fa1d6172640 1 -- 172.21.15.120:0/3834191863 >> v1:172.21.15.120:6789/0 conn(0x7fa1d013bfc0 legacy=0x7fa1d006f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.388+0000 7fa1d6172640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.388+0000 7fa1d6172640 1 monclient: found mon.noname-a 2024-02-06T21:41:38.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.388+0000 7fa1d6172640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.388+0000 7fa1d6172640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.388+0000 7fa1d6172640 1 -- 172.21.15.120:0/3834191863 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d006fc00 con 0x7fa1d00b12c0 2024-02-06T21:41:38.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.389+0000 7f7639764640 10 monclient(hunting): handle_auth_done global_id 4318 payload 274 2024-02-06T21:41:38.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.389+0000 7f7639764640 1 -- 172.21.15.120:0/64401410 >> v1:172.21.15.135:6789/0 conn(0x7f7634070cd0 legacy=0x7f763414f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.389+0000 7f7639764640 1 -- 172.21.15.120:0/64401410 >> v1:172.21.15.120:6789/0 conn(0x7f7634157880 legacy=0x7f7634159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.389+0000 7f7639764640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.389+0000 7f7639764640 1 monclient: found mon.noname-c 2024-02-06T21:41:38.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.389+0000 7f7639764640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.389+0000 7f7639764640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.389+0000 7f7639764640 1 -- 172.21.15.120:0/64401410 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763406f120 con 0x7f763408b2a0 2024-02-06T21:41:38.405 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.389+0000 7f1f977fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f1f9806f920 0x7f1f98143d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:53048/0 (socket says 172.21.15.120:53048) 2024-02-06T21:41:38.405 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.389+0000 7f1f977fe640 1 -- 172.21.15.120:0/2347109515 learned_addr learned my addr 172.21.15.120:0/2347109515 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:41:38.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.390+0000 7ff883ca5640 10 monclient: get_monmap_and_config 2024-02-06T21:41:38.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.390+0000 7ff883ca5640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.390+0000 7ff883ca5640 10 monclient: monmap: 2024-02-06T21:41:38.406 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.406 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.406 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:38.391662+0000 2024-02-06T21:41:38.407 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:38.391662+0000 2024-02-06T21:41:38.407 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.407 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.407 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:38.407 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:38.407 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:38.407 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:38.407 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:38.407 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.391+0000 7ff883ca5640 1 Processor -- start 2024-02-06T21:41:38.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.391+0000 7ff883ca5640 1 -- start start 2024-02-06T21:41:38.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.391+0000 7ff883ca5640 10 monclient: init 2024-02-06T21:41:38.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.392+0000 7ff883ca5640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.392+0000 7ff883ca5640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-02-06T21:41:38.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.392+0000 7ff883ca5640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff87c0b54b0 0x7ff87c0a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.392+0000 7ff883ca5640 10 monclient(hunting): picked mon.noname-c con 0x7ff87c0b54b0 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:38.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.392+0000 7ff883ca5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff87c13f830 0x7ff87c13fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.392+0000 7ff883ca5640 10 monclient(hunting): picked mon.noname-b con 0x7ff87c13f830 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.392+0000 7ff883ca5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff87c142680 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:38.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.392+0000 7ff883ca5640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87c143200 con 0x7ff87c142680 2024-02-06T21:41:38.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.392+0000 7ff883ca5640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.392+0000 7ff883ca5640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff87c0ab520 con 0x7ff87c0b54b0 2024-02-06T21:41:38.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.392+0000 7ff883ca5640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.392+0000 7ff883ca5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c0b5890 con 0x7ff87c13f830 2024-02-06T21:41:38.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.392+0000 7ff883ca5640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.392+0000 7ff883ca5640 10 monclient(hunting): authenticate will time out at 1328.729737s 2024-02-06T21:41:38.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.393+0000 7f1f97fff640 1 --2- 172.21.15.120:0/2347109515 >> v2:172.21.15.120:3300/0 conn(0x7f1f9806e7c0 0x7f1f9806eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.393+0000 7f1f97fff640 10 monclient(hunting): get_auth_request con 0x7f1f9806e7c0 auth_method 0 2024-02-06T21:41:38.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.393+0000 7f1f97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.393+0000 7f1f97fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.393+0000 7f1f97fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7ff882242640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7ff87c142680 0x7ff87c142af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.120:58866/0 (socket says 172.21.15.120:58866) 2024-02-06T21:41:38.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7ff882242640 1 -- 172.21.15.120:0/1298194425 learned_addr learned my addr 172.21.15.120:0/1298194425 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:41:38.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3eeeffd640 1 -- 172.21.15.120:0/3827070521 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0030610 con 0x7f3ef006f920 2024-02-06T21:41:38.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3eeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3eeeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:38.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3eeeffd640 10 monclient: dump: 2024-02-06T21:41:38.411 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.411 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.411 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.411 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.411 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.411 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.411 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.411 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.411 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.411 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3eeeffd640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3eeeffd640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3eeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3eeeffd640 1 -- 172.21.15.120:0/3827070521 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee0030770 con 0x7f3ef006f920 2024-02-06T21:41:38.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3eeeffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3eeeffd640 1 -- 172.21.15.120:0/3827070521 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0030b00 con 0x7f3ef006f920 2024-02-06T21:41:38.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3eeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3eeeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:38.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3eeeffd640 10 monclient: dump: 2024-02-06T21:41:38.412 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.412 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.412 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.412 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.413 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.413 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.413 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.413 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.413 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.413 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.413 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3ef67e6640 5 monclient: authenticate success, global_id 4286 2024-02-06T21:41:38.413 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3ef67e6640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:38.413 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f3ef67e6640 4 set_mon_vals no callback set 2024-02-06T21:41:38.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f1f97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f1f97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.394+0000 7f1f97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.395+0000 7f3ef67e6640 10 monclient: shutdown 2024-02-06T21:41:38.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.395+0000 7f3ef67e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.395+0000 7fa1d6172640 1 --2- 172.21.15.120:0/3834191863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1d00b12c0 0x7fa1d013ba80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc0097a0 tx=0x7fa1cc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f529f10b548c43b server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.395+0000 7f3ef67e6640 1 -- 172.21.15.120:0/3827070521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ef006f920 msgr2=0x7f3ef0143d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.395+0000 7f3ef67e6640 1 --2- 172.21.15.120:0/3827070521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ef006f920 0x7f3ef0143d20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0072730 tx=0x7f3ee002b960 comp rx=0 tx=0).stop 2024-02-06T21:41:38.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.395+0000 7f1f9c9ac640 1 --2- 172.21.15.120:0/2347109515 >> v2:172.21.15.135:3300/0 conn(0x7f1f98071110 0x7f1f98146fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.416 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.395+0000 7f3ef67e6640 1 -- 172.21.15.120:0/3827070521 shutdown_connections 2024-02-06T21:41:38.416 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.395+0000 7f3ef67e6640 1 --2- 172.21.15.120:0/3827070521 >> v2:172.21.15.135:3300/0 conn(0x7f3ef0071110 0x7f3ef0144260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.416 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.395+0000 7f3ef67e6640 1 --2- 172.21.15.120:0/3827070521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ef006f920 0x7f3ef0143d20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.416 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.395+0000 7f3ef67e6640 1 --2- 172.21.15.120:0/3827070521 >> v2:172.21.15.120:3300/0 conn(0x7f3ef006e7c0 0x7f3ef006eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.395+0000 7f3ef67e6640 1 -- 172.21.15.120:0/3827070521 >> 172.21.15.120:0/3827070521 conn(0x7f3ef00b0bc0 msgr2=0x7f3ef00b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.395+0000 7f3ef67e6640 1 -- 172.21.15.120:0/3827070521 shutdown_connections 2024-02-06T21:41:38.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.395+0000 7f3ef67e6640 1 -- 172.21.15.120:0/3827070521 wait complete. 2024-02-06T21:41:38.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.396+0000 7f3ef67e6640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.396+0000 7f3ef67e6640 10 monclient: monmap: 2024-02-06T21:41:38.417 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.417 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:38.417 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:38.417 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:38.417 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.417 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.417 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:38.417 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:38.417 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:38.418 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.396+0000 7ff881240640 1 --2- 172.21.15.120:0/1298194425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff87c13f830 0x7ff87c13fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.419 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.396+0000 7f1f9c9ac640 10 monclient(hunting): get_auth_request con 0x7f1f98071110 auth_method 0 2024-02-06T21:41:38.419 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.396+0000 7f1f9c9ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.419 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.396+0000 7f1f9c9ac640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.419 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.396+0000 7f1f9c9ac640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.419 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.396+0000 7f3ef67e6640 1 librados: starting msgr at 2024-02-06T21:41:38.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.397+0000 7ff881240640 10 monclient(hunting): get_auth_request con 0x7ff87c13f830 auth_method 0 2024-02-06T21:41:38.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.397+0000 7ff881240640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.397+0000 7ff881240640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.397+0000 7ff881240640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.397+0000 7f3ef67e6640 1 librados: starting objecter 2024-02-06T21:41:38.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.397+0000 7f3ef67e6640 1 Processor -- start 2024-02-06T21:41:38.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.397+0000 7f3ef67e6640 1 -- start start 2024-02-06T21:41:38.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.397+0000 7f3ef67e6640 1 librados: setting wanted keys 2024-02-06T21:41:38.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.397+0000 7f3ef67e6640 1 librados: calling monclient init 2024-02-06T21:41:38.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.397+0000 7f3ef67e6640 10 monclient: init 2024-02-06T21:41:38.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.397+0000 7f41174f1640 10 monclient: get_monmap_and_config 2024-02-06T21:41:38.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.397+0000 7f41174f1640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.397+0000 7f41174f1640 10 monclient: monmap: 2024-02-06T21:41:38.422 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.422 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.422 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:38.398991+0000 2024-02-06T21:41:38.422 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:38.398991+0000 2024-02-06T21:41:38.422 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.422 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.423 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:38.423 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:38.423 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:38.423 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:38.423 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:38.423 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.397+0000 7f7639764640 1 --2- 172.21.15.120:0/64401410 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f763408b2a0 0x7f763414f0c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f762c009e50 tx=0x7f762c02bfb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f7c7005c74bd5af server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.398+0000 7f41174f1640 1 Processor -- start 2024-02-06T21:41:38.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.398+0000 7f1f9c9ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.398+0000 7f1f9c9ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.398+0000 7f1f9c9ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.398+0000 7f3ef67e6640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.398+0000 7f3ef67e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-02-06T21:41:38.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.398+0000 7f3ef67e6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ef006e7c0 0x7f3ef0149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.426 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.398+0000 7f3ef67e6640 10 monclient(hunting): picked mon.noname-a con 0x7f3ef006e7c0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:38.426 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.398+0000 7f3ef67e6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ef006f920 0x7f3ef013e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.426 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.398+0000 7f3ef67e6640 10 monclient(hunting): picked mon.noname-c con 0x7f3ef006f920 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.426 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.398+0000 7f3ef67e6640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ef0071110 0x7f3ef013ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.426 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.398+0000 7f41174f1640 1 -- start start 2024-02-06T21:41:38.426 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.398+0000 7f41174f1640 10 monclient: init 2024-02-06T21:41:38.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7f3ef67e6640 10 monclient(hunting): picked mon.noname-b con 0x7f3ef0071110 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:38.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7f3ef67e6640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7f3ef67e6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef013fe30 con 0x7f3ef006e7c0 2024-02-06T21:41:38.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7f3ef67e6640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7f3ef67e6640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef00b33f0 con 0x7f3ef0071110 2024-02-06T21:41:38.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7f3ef67e6640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7f3ef67e6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef0140030 con 0x7f3ef006f920 2024-02-06T21:41:38.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7f3ef67e6640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7f3ef67e6640 10 monclient(hunting): authenticate will time out at 1328.736832s 2024-02-06T21:41:38.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7ff881240640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7ff881240640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7ff881240640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7f41174f1640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7f41174f1640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-02-06T21:41:38.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7f41174f1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f411006ae80 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:38.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.399+0000 7f41174f1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f411006fdf0 0x7f41100b1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.400+0000 7f41174f1640 10 monclient(hunting): picked mon.noname-b con 0x7f411006fdf0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.400+0000 7f41174f1640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4110071530 0x7f4110143c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.400+0000 7f41174f1640 10 monclient(hunting): picked mon.noname-a con 0x7f4110071530 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:38.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.400+0000 7f41174f1640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411006f510 con 0x7f411006ae80 2024-02-06T21:41:38.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.400+0000 7f41174f1640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.400+0000 7f41174f1640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f411006b260 con 0x7f4110071530 2024-02-06T21:41:38.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.400+0000 7f41174f1640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.400+0000 7f41174f1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4110072f00 con 0x7f411006fdf0 2024-02-06T21:41:38.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.400+0000 7f41174f1640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.400+0000 7f41174f1640 10 monclient(hunting): authenticate will time out at 1328.737943s 2024-02-06T21:41:38.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.401+0000 7ff881240640 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2024-02-06T21:41:38.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.401+0000 7ff881240640 1 -- 172.21.15.120:0/1298194425 >> v2:172.21.15.135:3300/0 conn(0x7ff87c0b54b0 msgr2=0x7ff87c0a9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:38.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.401+0000 7ff881240640 1 --2- 172.21.15.120:0/1298194425 >> v2:172.21.15.135:3300/0 conn(0x7ff87c0b54b0 0x7ff87c0a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.401+0000 7ff881240640 1 -- 172.21.15.120:0/1298194425 >> v1:172.21.15.135:6789/0 conn(0x7ff87c142680 legacy=0x7ff87c142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.401+0000 7ff881240640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.401+0000 7ff881240640 1 monclient: found mon.noname-b 2024-02-06T21:41:38.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.401+0000 7ff881240640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.433 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.401+0000 7ff881240640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.433 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.401+0000 7ff881240640 1 -- 172.21.15.120:0/1298194425 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c0ab720 con 0x7ff87c13f830 2024-02-06T21:41:38.433 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.403+0000 7f1f97fff640 10 monclient(hunting): handle_auth_done global_id 4320 payload 274 2024-02-06T21:41:38.433 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.403+0000 7f1f97fff640 1 -- 172.21.15.120:0/2347109515 >> v2:172.21.15.135:3300/0 conn(0x7f1f98071110 msgr2=0x7f1f98146fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:38.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.403+0000 7f1f97fff640 1 --2- 172.21.15.120:0/2347109515 >> v2:172.21.15.135:3300/0 conn(0x7f1f98071110 0x7f1f98146fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.403+0000 7f1f97fff640 1 -- 172.21.15.120:0/2347109515 >> v1:172.21.15.120:6789/0 conn(0x7f1f9806f920 legacy=0x7f1f98143d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.403+0000 7f1f97fff640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.403+0000 7f1f97fff640 1 monclient: found mon.noname-a 2024-02-06T21:41:38.435 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.403+0000 7f1f97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.403+0000 7f1f97fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.403+0000 7f1f97fff640 1 -- 172.21.15.120:0/2347109515 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f98144430 con 0x7f1f9806e7c0 2024-02-06T21:41:38.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.406+0000 7f3eef7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ef006f920 0x7f3ef013e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.406+0000 7f3eef7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ef006f920 0x7f3ef013e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:56958/0 (socket says 172.21.15.120:56958) 2024-02-06T21:41:38.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.406+0000 7f3eef7fe640 1 -- 172.21.15.120:0/1705276881 learned_addr learned my addr 172.21.15.120:0/1705276881 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.406+0000 7f3eef7fe640 10 monclient(hunting): get_auth_request con 0x7f3ef006f920 auth_method 0 2024-02-06T21:41:38.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.406+0000 7f3eef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.406+0000 7f3eef7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.406+0000 7f3eef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.406+0000 7f4114a8c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f411006fdf0 0x7f41100b1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f3eef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f3eef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f3eef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.439 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f762bfff640 1 -- 172.21.15.120:0/64401410 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c02cb30 con 0x7f763408b2a0 2024-02-06T21:41:38.439 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f762bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.439 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f762bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:38.439 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f762bfff640 10 monclient: dump: 2024-02-06T21:41:38.439 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.439 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.439 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.439 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.439 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.439 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.439 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.439 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.440 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.440 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f762bfff640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f762bfff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f762bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f763b9c8640 5 monclient: authenticate success, global_id 4318 2024-02-06T21:41:38.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f763b9c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:38.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f762bfff640 1 -- 172.21.15.120:0/64401410 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f762c02cc90 con 0x7f763408b2a0 2024-02-06T21:41:38.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f762bfff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f763b9c8640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:38.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f763b9c8640 4 set_mon_vals no callback set 2024-02-06T21:41:38.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f4114a8c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f411006fdf0 0x7f41100b1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:56970/0 (socket says 172.21.15.120:56970) 2024-02-06T21:41:38.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f4114a8c640 1 -- 172.21.15.120:0/202915895 learned_addr learned my addr 172.21.15.120:0/202915895 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f4114a8c640 10 monclient(hunting): get_auth_request con 0x7f411006fdf0 auth_method 0 2024-02-06T21:41:38.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f4114a8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f4114a8c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.407+0000 7f4114a8c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.408+0000 7f763b9c8640 10 monclient: shutdown 2024-02-06T21:41:38.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.408+0000 7f763b9c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.408+0000 7f763b9c8640 1 -- 172.21.15.120:0/64401410 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f763408b2a0 msgr2=0x7f763414f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.408+0000 7f763b9c8640 1 --2- 172.21.15.120:0/64401410 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f763408b2a0 0x7f763414f0c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f762c009e50 tx=0x7f762c02bfb0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.408+0000 7f763b9c8640 1 -- 172.21.15.120:0/64401410 shutdown_connections 2024-02-06T21:41:38.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.408+0000 7f763b9c8640 1 --2- 172.21.15.120:0/64401410 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f763408b2a0 0x7f763414f0c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.408+0000 7f763b9c8640 1 -- 172.21.15.120:0/64401410 >> 172.21.15.120:0/64401410 conn(0x7f76340b0cc0 msgr2=0x7f763406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.409+0000 7f763b9c8640 1 -- 172.21.15.120:0/64401410 shutdown_connections 2024-02-06T21:41:38.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.409+0000 7f763b9c8640 1 -- 172.21.15.120:0/64401410 wait complete. 2024-02-06T21:41:38.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.409+0000 7f763b9c8640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.409+0000 7f763b9c8640 10 monclient: monmap: 2024-02-06T21:41:38.443 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.443 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:38.443 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:38.443 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:38.444 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.444 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.444 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:38.444 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:38.444 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:38.444 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f763b9c8640 1 librados: starting msgr at 2024-02-06T21:41:38.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f763b9c8640 1 librados: starting objecter 2024-02-06T21:41:38.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f763b9c8640 1 Processor -- start 2024-02-06T21:41:38.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f3eef7fe640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2024-02-06T21:41:38.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f3eef7fe640 1 -- 172.21.15.120:0/1705276881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ef0071110 msgr2=0x7f3ef013ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.445 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f3eef7fe640 1 --2- 172.21.15.120:0/1705276881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ef0071110 0x7f3ef013ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.445 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f3eef7fe640 1 -- 172.21.15.120:0/1705276881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ef006e7c0 msgr2=0x7f3ef0149440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.445 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f3eef7fe640 1 --2- 172.21.15.120:0/1705276881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ef006e7c0 0x7f3ef0149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.445 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f3eef7fe640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.445 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f3eef7fe640 1 monclient: found mon.noname-c 2024-02-06T21:41:38.445 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f3eef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.445 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f3eef7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.445 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f3eef7fe640 1 -- 172.21.15.120:0/1705276881 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee00094d0 con 0x7f3ef006f920 2024-02-06T21:41:38.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f763b9c8640 1 -- start start 2024-02-06T21:41:38.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f763b9c8640 1 librados: setting wanted keys 2024-02-06T21:41:38.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f763b9c8640 1 librados: calling monclient init 2024-02-06T21:41:38.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.410+0000 7f763b9c8640 10 monclient: init 2024-02-06T21:41:38.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.411+0000 7f763b9c8640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.411+0000 7f763b9c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-02-06T21:41:38.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.411+0000 7f763b9c8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7634070cd0 0x7f7634155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.412+0000 7ff881240640 1 --2- 172.21.15.120:0/1298194425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff87c13f830 0x7ff87c13fc70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff86c0097a0 tx=0x7ff86c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=777d27ed159d9451 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.412+0000 7f763b9c8640 10 monclient(hunting): picked mon.noname-c con 0x7f7634070cd0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.413+0000 7f763b9c8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f763408b2a0 0x7f76340ab390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.413+0000 7f763b9c8640 10 monclient(hunting): picked mon.noname-a con 0x7f763408b2a0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:38.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.413+0000 7f763b9c8640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7634157880 0x7f76340a9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.413+0000 7f763b9c8640 10 monclient(hunting): picked mon.noname-b con 0x7f7634157880 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:38.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.413+0000 7f763b9c8640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.413+0000 7f763b9c8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76340aaa40 con 0x7f763408b2a0 2024-02-06T21:41:38.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.413+0000 7f1f97fff640 1 --2- 172.21.15.120:0/2347109515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f9806e7c0 0x7f1f9806eba0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1f9806e0f0 tx=0x7f1f8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15dd6d3620e48bb0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.413+0000 7f763b9c8640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.413+0000 7f763b9c8640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7634155c70 con 0x7f7634157880 2024-02-06T21:41:38.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.413+0000 7f763b9c8640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.413+0000 7f763b9c8640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7634156f40 con 0x7f7634070cd0 2024-02-06T21:41:38.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.413+0000 7f763b9c8640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.413+0000 7f763b9c8640 10 monclient(hunting): authenticate will time out at 1328.751433s 2024-02-06T21:41:38.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.414+0000 7f4114a8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.414+0000 7f4114a8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.414+0000 7f4114a8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.451 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.416+0000 7f3eef7fe640 1 --2- 172.21.15.120:0/1705276881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ef006f920 0x7f3ef013e500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0000c00 tx=0x7f3ee0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.451 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.416+0000 7f3eecff9640 10 client.?.objecter ms_handle_connect 0x7f3ef006f920 2024-02-06T21:41:38.451 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.416+0000 7f3eecff9640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:38.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.417+0000 7f4114a8c640 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2024-02-06T21:41:38.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.417+0000 7f4114a8c640 1 -- 172.21.15.120:0/202915895 >> v2:172.21.15.120:3300/0 conn(0x7f4110071530 msgr2=0x7f4110143c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:38.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.417+0000 7f4114a8c640 1 --2- 172.21.15.120:0/202915895 >> v2:172.21.15.120:3300/0 conn(0x7f4110071530 0x7f4110143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.417+0000 7f4114a8c640 1 -- 172.21.15.120:0/202915895 >> v1:172.21.15.120:6789/0 conn(0x7f411006ae80 legacy=0x7f41100727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:38.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.417+0000 7f4114a8c640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.417+0000 7f4114a8c640 1 monclient: found mon.noname-b 2024-02-06T21:41:38.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.417+0000 7f4114a8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.417+0000 7f4114a8c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.453 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.417+0000 7f4114a8c640 1 -- 172.21.15.120:0/202915895 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411006f0a0 con 0x7f411006fdf0 2024-02-06T21:41:38.453 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.417+0000 7f7639f65640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7634157880 0x7f76340a9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.453 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7f7639f65640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7634157880 0x7f76340a9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38676/0 (socket says 172.21.15.120:38676) 2024-02-06T21:41:38.453 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7f7639f65640 1 -- 172.21.15.120:0/784205979 learned_addr learned my addr 172.21.15.120:0/784205979 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.453 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7f7639f65640 10 monclient(hunting): get_auth_request con 0x7f7634157880 auth_method 0 2024-02-06T21:41:38.453 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7f7639f65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.453 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7f7639f65640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.453 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7f7639f65640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.454 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d5170640 1 -- 172.21.15.120:0/3834191863 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc031640 con 0x7fa1d00b12c0 2024-02-06T21:41:38.454 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d5170640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.454 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d5170640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:38.454 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d5170640 10 monclient: dump: 2024-02-06T21:41:38.454 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.454 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.454 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.454 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.454 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.454 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.455 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.455 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.455 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.455 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d5170640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d5170640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d5170640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d5170640 1 -- 172.21.15.120:0/3834191863 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1cc0317a0 con 0x7fa1d00b12c0 2024-02-06T21:41:38.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d5170640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d5170640 1 -- 172.21.15.120:0/3834191863 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc031b70 con 0x7fa1d00b12c0 2024-02-06T21:41:38.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d5170640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d5170640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:38.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d5170640 10 monclient: dump: 2024-02-06T21:41:38.455 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.456 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.456 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.456 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.456 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.456 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.456 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.456 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.456 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.456 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d83d6640 5 monclient: authenticate success, global_id 4314 2024-02-06T21:41:38.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d83d6640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:38.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.418+0000 7fa1d83d6640 4 set_mon_vals no callback set 2024-02-06T21:41:38.457 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.419+0000 7f7639f65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.457 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.419+0000 7f7639f65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.457 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.419+0000 7f7639f65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.459 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.419+0000 7fa1d83d6640 10 monclient: shutdown 2024-02-06T21:41:38.459 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.419+0000 7fa1d83d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.460 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.419+0000 7fa1d83d6640 1 -- 172.21.15.120:0/3834191863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1d00b12c0 msgr2=0x7fa1d013ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.460 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.419+0000 7fa1d83d6640 1 --2- 172.21.15.120:0/3834191863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1d00b12c0 0x7fa1d013ba80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc0097a0 tx=0x7fa1cc02b8f0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.460 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.420+0000 7fa1d83d6640 1 -- 172.21.15.120:0/3834191863 shutdown_connections 2024-02-06T21:41:38.460 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.420+0000 7fa1d83d6640 1 --2- 172.21.15.120:0/3834191863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1d00b12c0 0x7fa1d013ba80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.460 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.420+0000 7fa1d83d6640 1 -- 172.21.15.120:0/3834191863 >> 172.21.15.120:0/3834191863 conn(0x7fa1d00b07a0 msgr2=0x7fa1d00b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.460 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.420+0000 7fa1d83d6640 1 -- 172.21.15.120:0/3834191863 shutdown_connections 2024-02-06T21:41:38.460 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.420+0000 7fa1d83d6640 1 -- 172.21.15.120:0/3834191863 wait complete. 2024-02-06T21:41:38.460 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.421+0000 7fa1d83d6640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.460 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.421+0000 7fa1d83d6640 10 monclient: monmap: 2024-02-06T21:41:38.460 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.460 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:38.461 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:38.461 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:38.461 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.461 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.461 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:38.461 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:38.461 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:38.461 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.461 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.421+0000 7fa1d83d6640 1 librados: starting msgr at 2024-02-06T21:41:38.461 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.421+0000 7fa1d83d6640 1 librados: starting objecter 2024-02-06T21:41:38.462 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.421+0000 7fa1d83d6640 1 Processor -- start 2024-02-06T21:41:38.462 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.421+0000 7fa1d83d6640 1 -- start start 2024-02-06T21:41:38.462 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.421+0000 7fa1d83d6640 1 librados: setting wanted keys 2024-02-06T21:41:38.462 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.421+0000 7fa1d83d6640 1 librados: calling monclient init 2024-02-06T21:41:38.462 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.421+0000 7fa1d83d6640 10 monclient: init 2024-02-06T21:41:38.462 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.462 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-02-06T21:41:38.462 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1d0071900 0x7fa1d00ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 10 monclient(hunting): picked mon.noname-a con 0x7fa1d0071900 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:38.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1d00b12c0 0x7fa1d00aac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 10 monclient(hunting): picked mon.noname-b con 0x7fa1d00b12c0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:38.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1d013bfc0 0x7fa1d00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 10 monclient(hunting): picked mon.noname-c con 0x7fa1d013bfc0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d00abbe0 con 0x7fa1d0071900 2024-02-06T21:41:38.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d00ab770 con 0x7fa1d00b12c0 2024-02-06T21:41:38.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d00ab930 con 0x7fa1d013bfc0 2024-02-06T21:41:38.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.422+0000 7fa1d83d6640 10 monclient(hunting): authenticate will time out at 1328.760509s 2024-02-06T21:41:38.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.423+0000 7f4114a8c640 1 --2- 172.21.15.120:0/202915895 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f411006fdf0 0x7f41100b1360 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4100009870 tx=0x7f410002b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a7992123ce4c231 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.423+0000 7f7639f65640 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2024-02-06T21:41:38.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.423+0000 7f7639f65640 1 -- 172.21.15.120:0/784205979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7634070cd0 msgr2=0x7f7634155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.423+0000 7f7639f65640 1 --2- 172.21.15.120:0/784205979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7634070cd0 0x7f7634155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.423+0000 7f7639f65640 1 -- 172.21.15.120:0/784205979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f763408b2a0 msgr2=0x7f76340ab390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.423+0000 7f7639f65640 1 --2- 172.21.15.120:0/784205979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f763408b2a0 0x7f76340ab390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.423+0000 7f7639f65640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.423+0000 7f7639f65640 1 monclient: found mon.noname-b 2024-02-06T21:41:38.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.423+0000 7f7639f65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.423+0000 7f7639f65640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.423+0000 7f7639f65640 1 -- 172.21.15.120:0/784205979 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762c009b30 con 0x7f7634157880 2024-02-06T21:41:38.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.426+0000 7fa1d5971640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1d00b12c0 0x7fa1d00aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.427+0000 7f7639f65640 1 --2- 172.21.15.120:0/784205979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7634157880 0x7f76340a9600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f763000c3c0 tx=0x7f763000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.427+0000 7f7629ffb640 10 client.?.objecter ms_handle_connect 0x7f7634157880 2024-02-06T21:41:38.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.427+0000 7f7629ffb640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:38.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.427+0000 7fa1d5971640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1d00b12c0 0x7fa1d00aac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38678/0 (socket says 172.21.15.120:38678) 2024-02-06T21:41:38.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.427+0000 7fa1d5971640 1 -- 172.21.15.120:0/1429386313 learned_addr learned my addr 172.21.15.120:0/1429386313 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.427+0000 7fa1d5971640 10 monclient(hunting): get_auth_request con 0x7fa1d00b12c0 auth_method 0 2024-02-06T21:41:38.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.427+0000 7fa1d5971640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.427+0000 7fa1d5971640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.427+0000 7fa1d5971640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.468 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f96ffd640 1 -- 172.21.15.120:0/2347109515 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c031830 con 0x7f1f9806e7c0 2024-02-06T21:41:38.468 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.468 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f96ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:38.468 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f96ffd640 10 monclient: dump: 2024-02-06T21:41:38.468 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.468 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.468 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.468 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.468 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.468 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.468 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.468 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.468 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.468 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f96ffd640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f96ffd640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f96ffd640 1 -- 172.21.15.120:0/2347109515 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c031990 con 0x7f1f9806e7c0 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f96ffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f9e40f640 5 monclient: authenticate success, global_id 4320 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f96ffd640 1 -- 172.21.15.120:0/2347109515 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c031d20 con 0x7f1f9806e7c0 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f9e40f640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f9e40f640 4 set_mon_vals no callback set 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.429+0000 7f1f96ffd640 10 monclient: dump: 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.469 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.470 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.470 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.470 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.470 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.470 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.470 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.430+0000 7f1f9e40f640 10 monclient: shutdown 2024-02-06T21:41:38.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.430+0000 7f1f9e40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.430+0000 7f1f9e40f640 1 -- 172.21.15.120:0/2347109515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f9806e7c0 msgr2=0x7f1f9806eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.430+0000 7f1f9e40f640 1 --2- 172.21.15.120:0/2347109515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f9806e7c0 0x7f1f9806eba0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1f9806e0f0 tx=0x7f1f8c02b8d0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.430+0000 7f1f9e40f640 1 -- 172.21.15.120:0/2347109515 shutdown_connections 2024-02-06T21:41:38.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.430+0000 7f1f9e40f640 1 --2- 172.21.15.120:0/2347109515 >> v2:172.21.15.135:3300/0 conn(0x7f1f98071110 0x7f1f98146fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.430+0000 7f1f9e40f640 1 --2- 172.21.15.120:0/2347109515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f9806e7c0 0x7f1f9806eba0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.430+0000 7f1f9e40f640 1 -- 172.21.15.120:0/2347109515 >> 172.21.15.120:0/2347109515 conn(0x7f1f980b0bc0 msgr2=0x7f1f980b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.471 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.430+0000 7f1f9e40f640 1 -- 172.21.15.120:0/2347109515 shutdown_connections 2024-02-06T21:41:38.471 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.430+0000 7f1f9e40f640 1 -- 172.21.15.120:0/2347109515 wait complete. 2024-02-06T21:41:38.471 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.431+0000 7f1f9e40f640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.471 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.431+0000 7f1f9e40f640 10 monclient: monmap: 2024-02-06T21:41:38.471 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.471 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:38.471 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:38.471 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:38.471 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.471 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.471 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:38.471 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:38.471 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:38.471 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.472 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.431+0000 7f1f9e40f640 1 librados: starting msgr at 2024-02-06T21:41:38.472 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.431+0000 7f1f9e40f640 1 librados: starting objecter 2024-02-06T21:41:38.472 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.431+0000 7f1f9e40f640 1 Processor -- start 2024-02-06T21:41:38.473 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.431+0000 7ff880a3f640 1 -- 172.21.15.120:0/1298194425 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c02d040 con 0x7ff87c13f830 2024-02-06T21:41:38.473 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.431+0000 7ff880a3f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.473 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.431+0000 7ff880a3f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:38.473 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.431+0000 7ff880a3f640 10 monclient: dump: 2024-02-06T21:41:38.473 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.473 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.473 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.473 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.473 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.473 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.473 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.474 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.474 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.474 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7ff880a3f640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7ff880a3f640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7ff880a3f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7ff880a3f640 1 -- 172.21.15.120:0/1298194425 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86c031810 con 0x7ff87c13f830 2024-02-06T21:41:38.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7ff880a3f640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7ff880a3f640 1 -- 172.21.15.120:0/1298194425 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c02d040 con 0x7ff87c13f830 2024-02-06T21:41:38.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7ff880a3f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7ff880a3f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:38.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7ff880a3f640 10 monclient: dump: 2024-02-06T21:41:38.474 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.474 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.475 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.475 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.475 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.475 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.475 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.475 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.475 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.475 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7ff883ca5640 5 monclient: authenticate success, global_id 4289 2024-02-06T21:41:38.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7ff883ca5640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:38.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7ff883ca5640 4 set_mon_vals no callback set 2024-02-06T21:41:38.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f7629ffb640 1 -- 172.21.15.120:0/784205979 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76300052d0 con 0x7f7634157880 2024-02-06T21:41:38.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f7629ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f7629ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:38.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f7629ffb640 10 monclient: dump: 2024-02-06T21:41:38.476 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.476 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.476 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.476 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f7629ffb640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f7629ffb640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f7629ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f7629ffb640 1 -- 172.21.15.120:0/784205979 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7630005430 con 0x7f7634157880 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f7629ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f7629ffb640 1 -- 172.21.15.120:0/784205979 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7630007240 con 0x7f7634157880 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f7629ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f7629ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f7629ffb640 10 monclient: dump: 2024-02-06T21:41:38.477 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.478 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.478 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.478 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.478 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.478 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.478 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.478 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.478 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.478 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.478 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f763b9c8640 5 monclient: authenticate success, global_id 4333 2024-02-06T21:41:38.478 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f763b9c8640 10 monclient: _renew_subs 2024-02-06T21:41:38.478 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f763b9c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.478 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f763b9c8640 1 -- 172.21.15.120:0/784205979 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76340aaf00 con 0x7f7634157880 2024-02-06T21:41:38.479 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.431+0000 7fa1d5971640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.479 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.431+0000 7fa1d5971640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.479 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.431+0000 7fa1d5971640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.479 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7fa1d6973640 1 --2- 172.21.15.120:0/1429386313 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1d013bfc0 0x7fa1d00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.479 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.433+0000 7ff883ca5640 10 monclient: shutdown 2024-02-06T21:41:38.479 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.433+0000 7ff883ca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.479 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.433+0000 7ff883ca5640 1 -- 172.21.15.120:0/1298194425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff87c13f830 msgr2=0x7ff87c13fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.480 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.433+0000 7ff883ca5640 1 --2- 172.21.15.120:0/1298194425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff87c13f830 0x7ff87c13fc70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff86c0097a0 tx=0x7ff86c02b8b0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.480 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 1 -- start start 2024-02-06T21:41:38.480 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.433+0000 7ff883ca5640 1 -- 172.21.15.120:0/1298194425 shutdown_connections 2024-02-06T21:41:38.481 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.433+0000 7ff883ca5640 1 --2- 172.21.15.120:0/1298194425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff87c13f830 0x7ff87c13fc70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.481 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.433+0000 7ff883ca5640 1 --2- 172.21.15.120:0/1298194425 >> v2:172.21.15.135:3300/0 conn(0x7ff87c0b54b0 0x7ff87c0a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.481 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.433+0000 7ff883ca5640 1 -- 172.21.15.120:0/1298194425 >> 172.21.15.120:0/1298194425 conn(0x7ff87c0b8660 msgr2=0x7ff87c0b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.481 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 1 librados: setting wanted keys 2024-02-06T21:41:38.481 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 1 librados: calling monclient init 2024-02-06T21:41:38.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 10 monclient: init 2024-02-06T21:41:38.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 10 monclient: _add_conns ranks=[0,1,2] 2024-02-06T21:41:38.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.433+0000 7ff883ca5640 1 -- 172.21.15.120:0/1298194425 shutdown_connections 2024-02-06T21:41:38.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f9806e7c0 0x7f1f98146410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.433+0000 7ff883ca5640 1 -- 172.21.15.120:0/1298194425 wait complete. 2024-02-06T21:41:38.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 10 monclient(hunting): picked mon.noname-a con 0x7f1f9806e7c0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:38.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f9806f920 0x7f1f981417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f762b7fe640 4 set_mon_vals no callback set 2024-02-06T21:41:38.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.433+0000 7f763b9c8640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.484 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.433+0000 7f763b9c8640 10 monclient: _renew_subs 2024-02-06T21:41:38.484 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.433+0000 7f763b9c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.484 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.433+0000 7f763b9c8640 1 -- 172.21.15.120:0/784205979 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76340ab8d0 con 0x7f7634157880 2024-02-06T21:41:38.484 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.433+0000 7f763b9c8640 1 librados: init done 2024-02-06T21:41:38.485 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 10 monclient(hunting): picked mon.noname-b con 0x7f1f9806f920 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:38.485 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f98071110 0x7f1f98141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.485 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 10 monclient(hunting): picked mon.noname-c con 0x7f1f98071110 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.485 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.485 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f980b2d00 con 0x7f1f9806e7c0 2024-02-06T21:41:38.485 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.485 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9811b920 con 0x7f1f9806f920 2024-02-06T21:41:38.485 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9811bc00 con 0x7f1f98071110 2024-02-06T21:41:38.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.432+0000 7f1f9e40f640 10 monclient(hunting): authenticate will time out at 1328.770488s 2024-02-06T21:41:38.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7f3eecff9640 1 -- 172.21.15.120:0/1705276881 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0039c90 con 0x7f3ef006f920 2024-02-06T21:41:38.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7f3eecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7f3eecff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:38.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7f3eecff9640 10 monclient: dump: 2024-02-06T21:41:38.486 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.486 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.486 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.486 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.487 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.487 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.487 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.487 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.487 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.487 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.487 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7f3eecff9640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.487 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7f3eecff9640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.487 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7f3eecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.487 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7f3eecff9640 1 -- 172.21.15.120:0/1705276881 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee0039e30 con 0x7f3ef006f920 2024-02-06T21:41:38.487 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7f3eecff9640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.487 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7f3eecff9640 1 -- 172.21.15.120:0/1705276881 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0043640 con 0x7f3ef006f920 2024-02-06T21:41:38.487 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7f3eecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.487 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7f3eecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:38.487 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7f3eecff9640 10 monclient: dump: 2024-02-06T21:41:38.488 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.488 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.488 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.488 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.488 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.488 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.488 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.488 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.488 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.488 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.488 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7ff883ca5640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.489 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7fa1d6172640 1 --2- 172.21.15.120:0/1429386313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1d0071900 0x7fa1d00ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.489 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7ff883ca5640 10 monclient: monmap: 2024-02-06T21:41:38.489 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.489 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:38.489 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:38.489 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:38.489 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.490 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.490 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:38.490 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:38.490 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:38.490 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.490 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7f763b9c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.490 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7f763b9c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.491 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7f763b9c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.491 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7f763b9c8640 1 -- 172.21.15.120:0/784205979 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7634157080 con 0x7f7634157880 2024-02-06T21:41:38.491 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.434+0000 7f3eee7fc640 4 set_mon_vals no callback set 2024-02-06T21:41:38.491 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7f3ef67e6640 5 monclient: authenticate success, global_id 4292 2024-02-06T21:41:38.491 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7f3ef67e6640 10 monclient: _renew_subs 2024-02-06T21:41:38.491 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7fa1d5971640 10 monclient(hunting): handle_auth_done global_id 4336 payload 824 2024-02-06T21:41:38.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7fa1d5971640 1 -- 172.21.15.120:0/1429386313 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1d013bfc0 msgr2=0x7fa1d00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7fa1d5971640 1 --2- 172.21.15.120:0/1429386313 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1d013bfc0 0x7fa1d00ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7fa1d5971640 1 -- 172.21.15.120:0/1429386313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1d0071900 msgr2=0x7fa1d00ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7fa1d5971640 1 --2- 172.21.15.120:0/1429386313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1d0071900 0x7fa1d00ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7fa1d5971640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7fa1d5971640 1 monclient: found mon.noname-b 2024-02-06T21:41:38.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7fa1d5971640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7fa1d5971640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7fa1d5971640 1 -- 172.21.15.120:0/1429386313 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1cc009400 con 0x7fa1d00b12c0 2024-02-06T21:41:38.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7f3ef67e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7f3ef67e6640 1 -- 172.21.15.120:0/1705276881 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef0140450 con 0x7f3ef006f920 2024-02-06T21:41:38.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7ff883ca5640 1 librados: starting msgr at 2024-02-06T21:41:38.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7ff883ca5640 1 librados: starting objecter 2024-02-06T21:41:38.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7f3ef67e6640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7f3ef67e6640 10 monclient: _renew_subs 2024-02-06T21:41:38.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7f3ef67e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7f3ef67e6640 1 -- 172.21.15.120:0/1705276881 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef013c130 con 0x7f3ef006f920 2024-02-06T21:41:38.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7f3ef67e6640 1 librados: init done 2024-02-06T21:41:38.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7ff883ca5640 1 Processor -- start 2024-02-06T21:41:38.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7ff883ca5640 1 -- start start 2024-02-06T21:41:38.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7ff883ca5640 1 librados: setting wanted keys 2024-02-06T21:41:38.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7ff883ca5640 1 librados: calling monclient init 2024-02-06T21:41:38.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.435+0000 7ff883ca5640 10 monclient: init 2024-02-06T21:41:38.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T21:41:38.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff87c0b54b0 0x7ff87c139130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 10 monclient(hunting): picked mon.noname-c con 0x7ff87c0b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff87c13f830 0x7ff87c1366d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 10 monclient(hunting): picked mon.noname-b con 0x7ff87c13f830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:38.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff87c142680 0x7ff87c136c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 10 monclient(hunting): picked mon.noname-a con 0x7ff87c142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:38.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c1383e0 con 0x7ff87c142680 2024-02-06T21:41:38.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c1386c0 con 0x7ff87c13f830 2024-02-06T21:41:38.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c1389a0 con 0x7ff87c0b54b0 2024-02-06T21:41:38.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.436+0000 7ff883ca5640 10 monclient(hunting): authenticate will time out at 1328.774111s 2024-02-06T21:41:38.497 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.437+0000 7f3ef67e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.497 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.437+0000 7f3ef67e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.497 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.437+0000 7f3ef67e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.497 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.437+0000 7f3ef67e6640 1 -- 172.21.15.120:0/1705276881 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef01405e0 con 0x7f3ef006f920 2024-02-06T21:41:38.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.440+0000 7f1f97fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f9806e7c0 0x7f1f98146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.440+0000 7f1f97fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f9806e7c0 0x7f1f98146410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60280/0 (socket says 172.21.15.120:60280) 2024-02-06T21:41:38.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.441+0000 7f1f97fff640 1 -- 172.21.15.120:0/89737353 learned_addr learned my addr 172.21.15.120:0/89737353 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.441+0000 7f1f97fff640 10 monclient(hunting): get_auth_request con 0x7f1f9806e7c0 auth_method 0 2024-02-06T21:41:38.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.441+0000 7f1f97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.441+0000 7f1f97fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.441+0000 7f1f97fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f1f97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f1f97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f1f97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.500 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f4107fff640 1 -- 172.21.15.120:0/202915895 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4100031740 con 0x7f411006fdf0 2024-02-06T21:41:38.500 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f4107fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.500 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f4107fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:38.500 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f4107fff640 10 monclient: dump: 2024-02-06T21:41:38.500 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.500 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.500 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.500 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.501 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.501 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.501 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.501 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.501 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.501 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.501 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f4107fff640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.501 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f4107fff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.501 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f4107fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.501 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f4107fff640 1 -- 172.21.15.120:0/202915895 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41000318a0 con 0x7f411006fdf0 2024-02-06T21:41:38.501 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f4107fff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.501 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f4107fff640 1 -- 172.21.15.120:0/202915895 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4100031c30 con 0x7f411006fdf0 2024-02-06T21:41:38.501 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f4107fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.501 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f4107fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:38.502 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f4107fff640 10 monclient: dump: 2024-02-06T21:41:38.502 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.502 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.502 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.502 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.502 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.502 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.502 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.502 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.502 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.502 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.502 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f41174f1640 5 monclient: authenticate success, global_id 4295 2024-02-06T21:41:38.502 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f41174f1640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:38.502 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.442+0000 7f41174f1640 4 set_mon_vals no callback set 2024-02-06T21:41:38.503 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.443+0000 7f41174f1640 10 monclient: shutdown 2024-02-06T21:41:38.503 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.443+0000 7f41174f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.503 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.443+0000 7f41174f1640 1 -- 172.21.15.120:0/202915895 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f411006fdf0 msgr2=0x7f41100b1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.503 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.443+0000 7f41174f1640 1 --2- 172.21.15.120:0/202915895 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f411006fdf0 0x7f41100b1360 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4100009870 tx=0x7f410002b7e0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.503 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.444+0000 7f41174f1640 1 -- 172.21.15.120:0/202915895 shutdown_connections 2024-02-06T21:41:38.503 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.444+0000 7f41174f1640 1 --2- 172.21.15.120:0/202915895 >> v2:172.21.15.120:3300/0 conn(0x7f4110071530 0x7f4110143c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.503 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.444+0000 7f41174f1640 1 --2- 172.21.15.120:0/202915895 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f411006fdf0 0x7f41100b1360 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.503 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.444+0000 7f41174f1640 1 -- 172.21.15.120:0/202915895 >> 172.21.15.120:0/202915895 conn(0x7f41100b4f90 msgr2=0x7f41100b53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.503 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.444+0000 7f41174f1640 1 -- 172.21.15.120:0/202915895 shutdown_connections 2024-02-06T21:41:38.503 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.444+0000 7f41174f1640 1 -- 172.21.15.120:0/202915895 wait complete. 2024-02-06T21:41:38.503 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.445+0000 7f41174f1640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.503 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.445+0000 7f41174f1640 10 monclient: monmap: 2024-02-06T21:41:38.503 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.504 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:38.504 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:38.504 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:38.504 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.504 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.504 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:38.504 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:38.504 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:38.504 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.504 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.445+0000 7f1f97fff640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2024-02-06T21:41:38.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.445+0000 7f1f97fff640 1 -- 172.21.15.120:0/89737353 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f98071110 msgr2=0x7f1f98141cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.445+0000 7f1f97fff640 1 --2- 172.21.15.120:0/89737353 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f98071110 0x7f1f98141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.445+0000 7f1f97fff640 1 -- 172.21.15.120:0/89737353 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f9806f920 msgr2=0x7f1f981417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.445+0000 7f1f97fff640 1 --2- 172.21.15.120:0/89737353 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f9806f920 0x7f1f981417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.445+0000 7f1f97fff640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.445+0000 7f1f97fff640 1 monclient: found mon.noname-a 2024-02-06T21:41:38.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.445+0000 7f1f97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.445+0000 7f1f97fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.445+0000 7f1f97fff640 1 -- 172.21.15.120:0/89737353 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c009400 con 0x7f1f9806e7c0 2024-02-06T21:41:38.506 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.446+0000 7f41174f1640 1 librados: starting msgr at 2024-02-06T21:41:38.506 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.446+0000 7f41174f1640 1 librados: starting objecter 2024-02-06T21:41:38.506 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.446+0000 7f41174f1640 1 Processor -- start 2024-02-06T21:41:38.506 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.446+0000 7f41174f1640 1 -- start start 2024-02-06T21:41:38.506 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.446+0000 7f41174f1640 1 librados: setting wanted keys 2024-02-06T21:41:38.506 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.446+0000 7f41174f1640 1 librados: calling monclient init 2024-02-06T21:41:38.507 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.446+0000 7f41174f1640 10 monclient: init 2024-02-06T21:41:38.507 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.447+0000 7ff882242640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff87c142680 0x7ff87c136c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.447+0000 7f41174f1640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.447+0000 7f41174f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-02-06T21:41:38.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.447+0000 7f41174f1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f411006ae80 0x7f4110146f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.447+0000 7f41174f1640 10 monclient(hunting): picked mon.noname-a con 0x7f411006ae80 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:38.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.447+0000 7f41174f1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f411006fdf0 0x7f411013eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.447+0000 7f41174f1640 10 monclient(hunting): picked mon.noname-c con 0x7f411006fdf0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.448+0000 7f41174f1640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4110071530 0x7f411013f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.448+0000 7f41174f1640 10 monclient(hunting): picked mon.noname-b con 0x7f4110071530 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:38.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.448+0000 7f41174f1640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.448+0000 7f41174f1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4110138690 con 0x7f411006ae80 2024-02-06T21:41:38.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.448+0000 7f41174f1640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.448+0000 7f41174f1640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4110140200 con 0x7f4110071530 2024-02-06T21:41:38.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.448+0000 7f41174f1640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.448+0000 7f41174f1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411013e820 con 0x7f411006fdf0 2024-02-06T21:41:38.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.448+0000 7f41174f1640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.448+0000 7f41174f1640 10 monclient(hunting): authenticate will time out at 1328.785843s 2024-02-06T21:41:38.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.448+0000 7f1f97fff640 1 --2- 172.21.15.120:0/89737353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f9806e7c0 0x7f1f98146410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c000c00 tx=0x7f1f8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.449+0000 7f1f94ff9640 10 client.?.objecter ms_handle_connect 0x7f1f9806e7c0 2024-02-06T21:41:38.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.449+0000 7f1f94ff9640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:38.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.449+0000 7ff882242640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff87c142680 0x7ff87c136c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60284/0 (socket says 172.21.15.120:60284) 2024-02-06T21:41:38.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.449+0000 7ff882242640 1 -- 172.21.15.120:0/1040850199 learned_addr learned my addr 172.21.15.120:0/1040850199 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.449+0000 7ff882242640 10 monclient(hunting): get_auth_request con 0x7ff87c142680 auth_method 0 2024-02-06T21:41:38.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.449+0000 7ff882242640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.449+0000 7ff882242640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.449+0000 7ff882242640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.450+0000 7f411528d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f411006ae80 0x7f4110146f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.452+0000 7f411528d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f411006ae80 0x7f4110146f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60292/0 (socket says 172.21.15.120:60292) 2024-02-06T21:41:38.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.452+0000 7f411528d640 1 -- 172.21.15.120:0/3621380277 learned_addr learned my addr 172.21.15.120:0/3621380277 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.452+0000 7f411528d640 10 monclient(hunting): get_auth_request con 0x7f411006ae80 auth_method 0 2024-02-06T21:41:38.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.452+0000 7f411528d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.452+0000 7f411528d640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.452+0000 7f411528d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.453+0000 7ff882242640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.453+0000 7ff882242640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.453+0000 7ff882242640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.454+0000 7f411528d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.454+0000 7f411528d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.454+0000 7f411528d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.456+0000 7ff882242640 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2024-02-06T21:41:38.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.456+0000 7ff882242640 1 -- 172.21.15.120:0/1040850199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff87c0b54b0 msgr2=0x7ff87c139130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.456+0000 7ff882242640 1 --2- 172.21.15.120:0/1040850199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff87c0b54b0 0x7ff87c139130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.456+0000 7ff882242640 1 -- 172.21.15.120:0/1040850199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff87c13f830 msgr2=0x7ff87c1366d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.456+0000 7ff882242640 1 --2- 172.21.15.120:0/1040850199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff87c13f830 0x7ff87c1366d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.456+0000 7ff882242640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.456+0000 7ff882242640 1 monclient: found mon.noname-a 2024-02-06T21:41:38.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.456+0000 7ff882242640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.457+0000 7ff882242640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.457+0000 7ff882242640 1 -- 172.21.15.120:0/1040850199 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c009400 con 0x7ff87c142680 2024-02-06T21:41:38.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f94ff9640 1 -- 172.21.15.120:0/89737353 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c03a960 con 0x7f1f9806e7c0 2024-02-06T21:41:38.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f94ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f94ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:38.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f94ff9640 10 monclient: dump: 2024-02-06T21:41:38.515 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.515 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.515 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.515 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.515 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.515 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.516 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.516 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.516 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.516 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f94ff9640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f94ff9640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f94ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f9e40f640 5 monclient: authenticate success, global_id 4329 2024-02-06T21:41:38.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f9e40f640 10 monclient: _renew_subs 2024-02-06T21:41:38.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f9e40f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f9e40f640 1 -- 172.21.15.120:0/89737353 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f980b27e0 con 0x7f1f9806e7c0 2024-02-06T21:41:38.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f9e40f640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f9e40f640 10 monclient: _renew_subs 2024-02-06T21:41:38.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f9e40f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f9e40f640 1 -- 172.21.15.120:0/89737353 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f98142230 con 0x7f1f9806e7c0 2024-02-06T21:41:38.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f9e40f640 1 librados: init done 2024-02-06T21:41:38.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f94ff9640 1 -- 172.21.15.120:0/89737353 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c03aac0 con 0x7f1f9806e7c0 2024-02-06T21:41:38.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f94ff9640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f94ff9640 1 -- 172.21.15.120:0/89737353 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c03ae50 con 0x7f1f9806e7c0 2024-02-06T21:41:38.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f94ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f94ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:38.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.458+0000 7f1f94ff9640 10 monclient: dump: 2024-02-06T21:41:38.517 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.517 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.517 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.517 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.518 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.518 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.518 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.518 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.518 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.518 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.459+0000 7f1f967fc640 4 set_mon_vals no callback set 2024-02-06T21:41:38.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.459+0000 7f2941acf640 10 monclient: get_monmap_and_config 2024-02-06T21:41:38.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.459+0000 7f2941acf640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.459+0000 7f2941acf640 10 monclient: monmap: 2024-02-06T21:41:38.519 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.519 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.519 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:38.461038+0000 2024-02-06T21:41:38.519 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:38.461038+0000 2024-02-06T21:41:38.519 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.519 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.519 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:38.519 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:38.519 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:38.519 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:38.519 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:38.520 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.460+0000 7f1f9e40f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.460+0000 7f1f9e40f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.460+0000 7f1f9e40f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.460+0000 7f1f9e40f640 1 -- 172.21.15.120:0/89737353 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f78005080 con 0x7f1f9806e7c0 2024-02-06T21:41:38.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.460+0000 7f2941acf640 1 Processor -- start 2024-02-06T21:41:38.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.460+0000 7f2941acf640 1 -- start start 2024-02-06T21:41:38.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.460+0000 7f2941acf640 10 monclient: init 2024-02-06T21:41:38.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f2941acf640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f2941acf640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-02-06T21:41:38.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f2941acf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0b54b0 0x7f293c0a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f2941acf640 10 monclient(hunting): picked mon.noname-b con 0x7f293c0b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f2941acf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f293c13f830 0x7f293c13fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f2941acf640 10 monclient(hunting): picked mon.noname-a con 0x7f293c13f830 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:38.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f2941acf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f293c142680 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:38.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f2941acf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293c143200 con 0x7f293c142680 2024-02-06T21:41:38.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f2941acf640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f2941acf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f293c0ab520 con 0x7f293c13f830 2024-02-06T21:41:38.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f2941acf640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f2941acf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293c0b5890 con 0x7f293c0b54b0 2024-02-06T21:41:38.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f2941acf640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f2941acf640 10 monclient(hunting): authenticate will time out at 1328.799078s 2024-02-06T21:41:38.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f411528d640 10 monclient(hunting): handle_auth_done global_id 4335 payload 824 2024-02-06T21:41:38.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f411528d640 1 -- 172.21.15.120:0/3621380277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f411006fdf0 msgr2=0x7f411013eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f411528d640 1 --2- 172.21.15.120:0/3621380277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f411006fdf0 0x7f411013eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f411528d640 1 -- 172.21.15.120:0/3621380277 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4110071530 msgr2=0x7f411013f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f411528d640 1 --2- 172.21.15.120:0/3621380277 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4110071530 0x7f411013f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f411528d640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f411528d640 1 monclient: found mon.noname-a 2024-02-06T21:41:38.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f411528d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f411528d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.461+0000 7f411528d640 1 -- 172.21.15.120:0/3621380277 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41000094d0 con 0x7f411006ae80 2024-02-06T21:41:38.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.466+0000 7f411528d640 1 --2- 172.21.15.120:0/3621380277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f411006ae80 0x7f4110146f80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f410c00c440 tx=0x7f410c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.466+0000 7f4105ffb640 10 client.?.objecter ms_handle_connect 0x7f411006ae80 2024-02-06T21:41:38.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.466+0000 7f4105ffb640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:38.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.513+0000 7f10f32d3640 10 monclient: get_monmap_and_config 2024-02-06T21:41:38.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.513+0000 7f10f32d3640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.513+0000 7f10f32d3640 10 monclient: monmap: 2024-02-06T21:41:38.527 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.527 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.528 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:38.514788+0000 2024-02-06T21:41:38.528 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:38.514788+0000 2024-02-06T21:41:38.528 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.528 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.528 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:38.528 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:38.528 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:38.528 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:38.528 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:38.528 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.514+0000 7f10f32d3640 1 Processor -- start 2024-02-06T21:41:38.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.514+0000 7f10f32d3640 1 -- start start 2024-02-06T21:41:38.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.514+0000 7f10f32d3640 10 monclient: init 2024-02-06T21:41:38.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.514+0000 7f10f32d3640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.514+0000 7f10f32d3640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-02-06T21:41:38.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.514+0000 7f10f32d3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10ec0b54b0 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:38.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.515+0000 7f10f32d3640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f10ec13ff40 0x7f10ec0a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.515+0000 7f10f32d3640 10 monclient(hunting): picked mon.noname-c con 0x7f10ec13ff40 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:38.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.515+0000 7f10f32d3640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f10ec142680 0x7f10ec142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.515+0000 7f10f32d3640 10 monclient(hunting): picked mon.noname-a con 0x7f10ec142680 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:38.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.515+0000 7f10f32d3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec0ab720 con 0x7f10ec0b54b0 2024-02-06T21:41:38.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.515+0000 7f10f32d3640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.515+0000 7f10f32d3640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10ec0ab520 con 0x7f10ec142680 2024-02-06T21:41:38.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.515+0000 7f10f32d3640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.515+0000 7f10f32d3640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10ec0b5890 con 0x7f10ec13ff40 2024-02-06T21:41:38.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.515+0000 7f10f32d3640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.515+0000 7f10f32d3640 10 monclient(hunting): authenticate will time out at 1328.852795s 2024-02-06T21:41:38.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.658+0000 7f7629ffb640 1 -- 172.21.15.120:0/784205979 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f76300073a0 con 0x7f7634157880 2024-02-06T21:41:38.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.658+0000 7f7629ffb640 1 --2- 172.21.15.120:0/784205979 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f761806d220 0x7f761806f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.658+0000 7f7639764640 1 --2- 172.21.15.120:0/784205979 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f761806d220 0x7f761806f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.658+0000 7f7639764640 10 monclient: get_auth_request con 0x7f761806d220 auth_method 0 2024-02-06T21:41:38.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.659+0000 7f7639764640 1 --2- 172.21.15.120:0/784205979 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f761806d220 0x7f761806f6e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f762c035000 tx=0x7f762c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.659+0000 7f7629ffb640 10 client.4333.objecter ms_handle_connect 0x7f761806d220 2024-02-06T21:41:38.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.702+0000 7f7629ffb640 1 -- 172.21.15.120:0/784205979 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7630007650 con 0x7f7634157880 2024-02-06T21:41:38.704 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.702+0000 7f7629ffb640 10 client.4333.objecter ms_dispatch 0x7f7634000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:38.704 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.702+0000 7f7629ffb640 3 client.4333.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:38.704 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.702+0000 7f7629ffb640 3 client.4333.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:38.704 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.702+0000 7f7629ffb640 20 client.4333.objecter dump_active .. 0 homeless 2024-02-06T21:41:38.706 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.704+0000 7f293b7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0b54b0 0x7f293c0a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.707 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.706+0000 7f293b7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0b54b0 0x7f293c0a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57032/0 (socket says 172.21.15.120:57032) 2024-02-06T21:41:38.707 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.706+0000 7f293b7fe640 1 -- 172.21.15.120:0/4161258038 learned_addr learned my addr 172.21.15.120:0/4161258038 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.707 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.706+0000 7f293b7fe640 10 monclient(hunting): get_auth_request con 0x7f293c0b54b0 auth_method 0 2024-02-06T21:41:38.707 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.706+0000 7f293b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.707 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.706+0000 7f293b7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.707 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.706+0000 7f293b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.714 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.713+0000 7f7629ffb640 1 -- 172.21.15.120:0/784205979 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f76300546c0 con 0x7f7634157880 2024-02-06T21:41:38.714 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.713+0000 7f7629ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.715 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.713+0000 7f7629ffb640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:38.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.714+0000 7fa1d5971640 1 --2- 172.21.15.120:0/1429386313 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1d00b12c0 0x7fa1d00aac90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0001eb0 tx=0x7fa1c000c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.714+0000 7fa1c6ffd640 10 client.?.objecter ms_handle_connect 0x7fa1d00b12c0 2024-02-06T21:41:38.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.714+0000 7fa1c6ffd640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:38.717 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.715+0000 7f10f086e640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f10ec13ff40 0x7f10ec0a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.717 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.716+0000 7f10f086e640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f10ec13ff40 0x7f10ec0a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38724/0 (socket says 172.21.15.120:38724) 2024-02-06T21:41:38.717 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.716+0000 7f10f086e640 1 -- 172.21.15.120:0/2790733828 learned_addr learned my addr 172.21.15.120:0/2790733828 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.717 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.716+0000 7f10f086e640 10 monclient(hunting): get_auth_request con 0x7f10ec13ff40 auth_method 0 2024-02-06T21:41:38.718 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.716+0000 7f10f086e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.718 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.716+0000 7f10f086e640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.718 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.716+0000 7f10f086e640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.721 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.719+0000 7f10f086e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.721 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.719+0000 7f10f086e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.721 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.719+0000 7f10f086e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.722 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.719+0000 7f3eecff9640 1 -- 172.21.15.120:0/1705276881 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3ee00437a0 con 0x7f3ef006f920 2024-02-06T21:41:38.722 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.720+0000 7f3eecff9640 1 --2- 172.21.15.120:0/1705276881 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f3edc06d220 0x7f3edc06f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.722 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.720+0000 7f3eeffff640 1 --2- 172.21.15.120:0/1705276881 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f3edc06d220 0x7f3edc06f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.722 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.720+0000 7f3eeffff640 10 monclient: get_auth_request con 0x7f3edc06d220 auth_method 0 2024-02-06T21:41:38.722 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.720+0000 7f3eecff9640 10 client.4292.objecter ms_handle_connect 0x7f3edc06d220 2024-02-06T21:41:38.722 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.720+0000 7f3eeffff640 1 --2- 172.21.15.120:0/1705276881 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f3edc06d220 0x7f3edc06f6e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3ee80070b0 tx=0x7f3ee8007040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.723 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.722+0000 7f293b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.724 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.722+0000 7f293b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.724 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.722+0000 7f293b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.724 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.723+0000 7ff882242640 1 --2- 172.21.15.120:0/1040850199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff87c142680 0x7ff87c136c10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff87400c9f0 tx=0x7ff87400ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.724 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.723+0000 7ff8727fc640 10 client.?.objecter ms_handle_connect 0x7ff87c142680 2024-02-06T21:41:38.724 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.723+0000 7ff8727fc640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:38.725 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.723+0000 7f1f94ff9640 1 -- 172.21.15.120:0/89737353 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1f8c0443f0 con 0x7f1f9806e7c0 2024-02-06T21:41:38.725 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.723+0000 7f1f94ff9640 1 --2- 172.21.15.120:0/89737353 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1f8406d1a0 0x7f1f8406f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.725 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.724+0000 7f1f977fe640 1 --2- 172.21.15.120:0/89737353 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1f8406d1a0 0x7f1f8406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.725 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.724+0000 7f1f977fe640 10 monclient: get_auth_request con 0x7f1f8406d1a0 auth_method 0 2024-02-06T21:41:38.726 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.724+0000 7f1f94ff9640 10 client.4329.objecter ms_handle_connect 0x7f1f8406d1a0 2024-02-06T21:41:38.726 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.724+0000 7f1f977fe640 1 --2- 172.21.15.120:0/89737353 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1f8406d1a0 0x7f1f8406f660 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1f980b22b0 tx=0x7f1f88007a40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.727 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.725+0000 7f10f086e640 10 monclient(hunting): handle_auth_done global_id 4339 payload 274 2024-02-06T21:41:38.727 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.725+0000 7f10f086e640 1 -- 172.21.15.120:0/2790733828 >> v2:172.21.15.120:3300/0 conn(0x7f10ec142680 msgr2=0x7f10ec142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:38.727 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.725+0000 7f10f086e640 1 --2- 172.21.15.120:0/2790733828 >> v2:172.21.15.120:3300/0 conn(0x7f10ec142680 0x7f10ec142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.727 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.725+0000 7f10f086e640 1 -- 172.21.15.120:0/2790733828 >> v1:172.21.15.120:6789/0 conn(0x7f10ec0b54b0 legacy=0x7f10ec13f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:38.727 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.725+0000 7f10f086e640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.727 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.725+0000 7f10f086e640 1 monclient: found mon.noname-c 2024-02-06T21:41:38.727 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.725+0000 7f10f086e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.727 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.725+0000 7f10f086e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.728 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.725+0000 7f10f086e640 1 -- 172.21.15.120:0/2790733828 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec0b4800 con 0x7f10ec13ff40 2024-02-06T21:41:38.729 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1c6ffd640 1 -- 172.21.15.120:0/1429386313 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c00052d0 con 0x7fa1d00b12c0 2024-02-06T21:41:38.729 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.729 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:38.729 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1c6ffd640 10 monclient: dump: 2024-02-06T21:41:38.729 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.729 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.729 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.730 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.730 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.730 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.730 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.730 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.730 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.730 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.730 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1c6ffd640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.730 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1c6ffd640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.730 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.730 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1c6ffd640 1 -- 172.21.15.120:0/1429386313 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1c0005430 con 0x7fa1d00b12c0 2024-02-06T21:41:38.730 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1d83d6640 5 monclient: authenticate success, global_id 4336 2024-02-06T21:41:38.730 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.730 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1c6ffd640 1 -- 172.21.15.120:0/1429386313 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0003830 con 0x7fa1d00b12c0 2024-02-06T21:41:38.730 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:38.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1c6ffd640 10 monclient: dump: 2024-02-06T21:41:38.731 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.731 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.731 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.731 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.731 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.731 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.731 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.731 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.731 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.731 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1d83d6640 10 monclient: _renew_subs 2024-02-06T21:41:38.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1d496f640 4 set_mon_vals no callback set 2024-02-06T21:41:38.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1d83d6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1d83d6640 1 -- 172.21.15.120:0/1429386313 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d00ae1f0 con 0x7fa1d00b12c0 2024-02-06T21:41:38.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1d83d6640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1d83d6640 10 monclient: _renew_subs 2024-02-06T21:41:38.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1d83d6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1d83d6640 1 -- 172.21.15.120:0/1429386313 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d01db640 con 0x7fa1d00b12c0 2024-02-06T21:41:38.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.726+0000 7fa1d83d6640 1 librados: init done 2024-02-06T21:41:38.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.727+0000 7fa1d83d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.727+0000 7fa1d83d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.727+0000 7fa1d83d6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.727+0000 7fa1d83d6640 1 -- 172.21.15.120:0/1429386313 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d00ac5b0 con 0x7fa1d00b12c0 2024-02-06T21:41:38.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.729+0000 7f10f086e640 1 --2- 172.21.15.120:0/2790733828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10ec13ff40 0x7f10ec0a9860 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f10ec06deb0 tx=0x7f10dc02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcb5ca5a131ca304 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.735 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.733+0000 7f3eecff9640 1 -- 172.21.15.120:0/1705276881 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3ee003e070 con 0x7f3ef006f920 2024-02-06T21:41:38.735 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.733+0000 7f3eecff9640 10 client.4292.objecter ms_dispatch 0x7f3ef0000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:38.735 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.733+0000 7f3eecff9640 3 client.4292.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:38.735 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.733+0000 7f3eecff9640 3 client.4292.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:38.735 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.733+0000 7f3eecff9640 20 client.4292.objecter dump_active .. 0 homeless 2024-02-06T21:41:38.735 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.733+0000 7f3eecff9640 1 -- 172.21.15.120:0/1705276881 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3ee00d98c0 con 0x7f3ef006f920 2024-02-06T21:41:38.735 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.733+0000 7f3eecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.735 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.733+0000 7f3eecff9640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:38.736 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.735+0000 7f293b7fe640 10 monclient(hunting): handle_auth_done global_id 4298 payload 274 2024-02-06T21:41:38.736 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.735+0000 7f293b7fe640 1 -- 172.21.15.120:0/4161258038 >> v2:172.21.15.120:3300/0 conn(0x7f293c13f830 msgr2=0x7f293c13fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:38.737 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.735+0000 7f293b7fe640 1 --2- 172.21.15.120:0/4161258038 >> v2:172.21.15.120:3300/0 conn(0x7f293c13f830 0x7f293c13fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.737 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.735+0000 7f293b7fe640 1 -- 172.21.15.120:0/4161258038 >> v1:172.21.15.120:6789/0 conn(0x7f293c142680 legacy=0x7f293c142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:38.737 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.735+0000 7f293b7fe640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.737 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.735+0000 7f293b7fe640 1 monclient: found mon.noname-b 2024-02-06T21:41:38.737 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.735+0000 7f293b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.737 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.735+0000 7f293b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.737 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.735+0000 7f293b7fe640 1 -- 172.21.15.120:0/4161258038 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293c0ab720 con 0x7f293c0b54b0 2024-02-06T21:41:38.740 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.739+0000 7f293b7fe640 1 --2- 172.21.15.120:0/4161258038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0b54b0 0x7f293c0a9860 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f292c009870 tx=0x7f292c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f8f085b10a00172 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.742 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.741+0000 7f1f94ff9640 1 -- 172.21.15.120:0/89737353 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1f8c07ba30 con 0x7f1f9806e7c0 2024-02-06T21:41:38.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.741+0000 7f1f94ff9640 10 client.4329.objecter ms_dispatch 0x7f1f98000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:38.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.741+0000 7f1f94ff9640 3 client.4329.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:38.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.741+0000 7f1f94ff9640 3 client.4329.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:38.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.741+0000 7f1f94ff9640 20 client.4329.objecter dump_active .. 0 homeless 2024-02-06T21:41:38.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.741+0000 7f1f94ff9640 1 -- 172.21.15.120:0/89737353 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1f8c0775d0 con 0x7f1f9806e7c0 2024-02-06T21:41:38.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.741+0000 7f1f94ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.741+0000 7f1f94ff9640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:38.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f293a7fc640 1 -- 172.21.15.120:0/4161258038 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c031640 con 0x7f293c0b54b0 2024-02-06T21:41:38.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f293a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f293a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:38.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f293a7fc640 10 monclient: dump: 2024-02-06T21:41:38.746 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f293a7fc640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f293a7fc640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f293a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f293a7fc640 1 -- 172.21.15.120:0/4161258038 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f292c0317a0 con 0x7f293c0b54b0 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f293a7fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f293a7fc640 1 -- 172.21.15.120:0/4161258038 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c031b30 con 0x7f293c0b54b0 2024-02-06T21:41:38.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f293a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.748 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f293a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:38.748 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f293a7fc640 10 monclient: dump: 2024-02-06T21:41:38.748 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.748 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.748 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.748 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.748 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.748 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.748 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.748 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.748 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.748 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.748 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f2941acf640 5 monclient: authenticate success, global_id 4298 2024-02-06T21:41:38.748 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f2941acf640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:38.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.744+0000 7f2941acf640 4 set_mon_vals no callback set 2024-02-06T21:41:38.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.745+0000 7f2941acf640 10 monclient: shutdown 2024-02-06T21:41:38.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.745+0000 7f2941acf640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.745+0000 7f2941acf640 1 -- 172.21.15.120:0/4161258038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0b54b0 msgr2=0x7f293c0a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.745+0000 7f2941acf640 1 --2- 172.21.15.120:0/4161258038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0b54b0 0x7f293c0a9860 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f292c009870 tx=0x7f292c02b8f0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.746+0000 7f2941acf640 1 -- 172.21.15.120:0/4161258038 shutdown_connections 2024-02-06T21:41:38.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.746+0000 7f2941acf640 1 --2- 172.21.15.120:0/4161258038 >> v2:172.21.15.120:3300/0 conn(0x7f293c13f830 0x7f293c13fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.746+0000 7f2941acf640 1 --2- 172.21.15.120:0/4161258038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c0b54b0 0x7f293c0a9860 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.746+0000 7f2941acf640 1 -- 172.21.15.120:0/4161258038 >> 172.21.15.120:0/4161258038 conn(0x7f293c0b8660 msgr2=0x7f293c0b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.746+0000 7f2941acf640 1 -- 172.21.15.120:0/4161258038 shutdown_connections 2024-02-06T21:41:38.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.746+0000 7f2941acf640 1 -- 172.21.15.120:0/4161258038 wait complete. 2024-02-06T21:41:38.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.747+0000 7f2941acf640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.747+0000 7f2941acf640 10 monclient: monmap: 2024-02-06T21:41:38.752 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.752 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:38.752 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:38.752 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:38.752 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.752 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.752 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:38.752 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:38.752 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:38.753 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.748+0000 7f2941acf640 1 librados: starting msgr at 2024-02-06T21:41:38.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.748+0000 7f2941acf640 1 librados: starting objecter 2024-02-06T21:41:38.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.748+0000 7f2941acf640 1 Processor -- start 2024-02-06T21:41:38.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.748+0000 7f2941acf640 1 -- start start 2024-02-06T21:41:38.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.748+0000 7f2941acf640 1 librados: setting wanted keys 2024-02-06T21:41:38.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.748+0000 7f2941acf640 1 librados: calling monclient init 2024-02-06T21:41:38.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.748+0000 7f2941acf640 10 monclient: init 2024-02-06T21:41:38.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:38.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T21:41:38.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f293c0b54b0 0x7f293c135c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 10 monclient(hunting): picked mon.noname-b con 0x7f293c0b54b0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:38.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f293c13f830 0x7f293c138760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 10 monclient(hunting): picked mon.noname-a con 0x7f293c13f830 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:38.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c142680 0x7f293c138ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 10 monclient(hunting): picked mon.noname-c con 0x7f293c142680 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:38.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c137aa0 con 0x7f293c13f830 2024-02-06T21:41:38.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c137d80 con 0x7f293c0b54b0 2024-02-06T21:41:38.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:38.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293c138060 con 0x7f293c142680 2024-02-06T21:41:38.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:38.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.749+0000 7f2941acf640 10 monclient(hunting): authenticate will time out at 1329.087084s 2024-02-06T21:41:38.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.750+0000 7f293bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c142680 0x7f293c138ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.750+0000 7f293bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c142680 0x7f293c138ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57042/0 (socket says 172.21.15.120:57042) 2024-02-06T21:41:38.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.750+0000 7f293bfff640 1 -- 172.21.15.120:0/1062616246 learned_addr learned my addr 172.21.15.120:0/1062616246 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:38.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.750+0000 7f293bfff640 10 monclient(hunting): get_auth_request con 0x7f293c142680 auth_method 0 2024-02-06T21:41:38.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.750+0000 7f293bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:38.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.750+0000 7f293bfff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:38.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.750+0000 7f293bfff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:38.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.751+0000 7f293bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:38.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.751+0000 7f293bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:38.756 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.751+0000 7f293bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:38.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.755+0000 7f293bfff640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-02-06T21:41:38.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.755+0000 7f293bfff640 1 -- 172.21.15.120:0/1062616246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f293c0b54b0 msgr2=0x7f293c135c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.755+0000 7f293bfff640 1 --2- 172.21.15.120:0/1062616246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f293c0b54b0 0x7f293c135c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.755+0000 7f293bfff640 1 -- 172.21.15.120:0/1062616246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f293c13f830 msgr2=0x7f293c138760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.755+0000 7f293bfff640 1 --2- 172.21.15.120:0/1062616246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f293c13f830 0x7f293c138760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.755+0000 7f293bfff640 10 monclient: _finish_hunting 0 2024-02-06T21:41:38.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.755+0000 7f293bfff640 1 monclient: found mon.noname-c 2024-02-06T21:41:38.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.755+0000 7f293bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:38.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.755+0000 7f293bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.755+0000 7f293bfff640 1 -- 172.21.15.120:0/1062616246 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292c0094d0 con 0x7f293c142680 2024-02-06T21:41:38.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.759+0000 7f293bfff640 1 --2- 172.21.15.120:0/1062616246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c142680 0x7f293c138ca0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f293400b990 tx=0x7f293400be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.759+0000 7f2923fff640 10 client.?.objecter ms_handle_connect 0x7f293c142680 2024-02-06T21:41:38.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.759+0000 7f2923fff640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:38.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff8727fc640 1 -- 172.21.15.120:0/1040850199 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87400ab70 con 0x7ff87c142680 2024-02-06T21:41:38.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff8727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff8727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:38.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff8727fc640 10 monclient: dump: 2024-02-06T21:41:38.763 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.763 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.763 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.763 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.763 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.763 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.763 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.763 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.763 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.763 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff8727fc640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff8727fc640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff8727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff883ca5640 5 monclient: authenticate success, global_id 4332 2024-02-06T21:41:38.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff8727fc640 1 -- 172.21.15.120:0/1040850199 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87400acd0 con 0x7ff87c142680 2024-02-06T21:41:38.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff883ca5640 10 monclient: _renew_subs 2024-02-06T21:41:38.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff883ca5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff883ca5640 1 -- 172.21.15.120:0/1040850199 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87c139730 con 0x7ff87c142680 2024-02-06T21:41:38.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff8727fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff8727fc640 1 -- 172.21.15.120:0/1040850199 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874012600 con 0x7ff87c142680 2024-02-06T21:41:38.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff8727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff8727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:38.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff8727fc640 10 monclient: dump: 2024-02-06T21:41:38.764 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.764 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.764 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.764 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.764 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.765 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.765 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.765 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.765 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.765 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff883ca5640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff883ca5640 10 monclient: _renew_subs 2024-02-06T21:41:38.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff883ca5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff883ca5640 1 -- 172.21.15.120:0/1040850199 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87c1377d0 con 0x7ff87c142680 2024-02-06T21:41:38.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff883ca5640 1 librados: init done 2024-02-06T21:41:38.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.760+0000 7ff873fff640 4 set_mon_vals no callback set 2024-02-06T21:41:38.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.761+0000 7ff883ca5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.761+0000 7ff883ca5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.761+0000 7ff883ca5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.761+0000 7ff883ca5640 1 -- 172.21.15.120:0/1040850199 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff87c140130 con 0x7ff87c142680 2024-02-06T21:41:38.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f4105ffb640 1 -- 172.21.15.120:0/3621380277 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c005480 con 0x7f411006ae80 2024-02-06T21:41:38.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f4105ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f4105ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:38.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f4105ffb640 10 monclient: dump: 2024-02-06T21:41:38.768 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.768 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.768 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.768 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.769 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.769 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.769 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.769 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.769 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.769 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f4105ffb640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f4105ffb640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f4105ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f4105ffb640 1 -- 172.21.15.120:0/3621380277 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410c0035f0 con 0x7f411006ae80 2024-02-06T21:41:38.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f4105ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f4105ffb640 1 -- 172.21.15.120:0/3621380277 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c0039c0 con 0x7f411006ae80 2024-02-06T21:41:38.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f4105ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f4105ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:38.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f4105ffb640 10 monclient: dump: 2024-02-06T21:41:38.770 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.770 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.770 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.770 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.770 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.770 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.770 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.770 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.770 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.770 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f41077fe640 4 set_mon_vals no callback set 2024-02-06T21:41:38.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f41174f1640 5 monclient: authenticate success, global_id 4335 2024-02-06T21:41:38.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f41174f1640 10 monclient: _renew_subs 2024-02-06T21:41:38.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f41174f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f41174f1640 1 -- 172.21.15.120:0/3621380277 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411013ed10 con 0x7f411006ae80 2024-02-06T21:41:38.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f41174f1640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f41174f1640 10 monclient: _renew_subs 2024-02-06T21:41:38.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f41174f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f41174f1640 1 -- 172.21.15.120:0/3621380277 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4110140a10 con 0x7f411006ae80 2024-02-06T21:41:38.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.764+0000 7f41174f1640 1 librados: init done 2024-02-06T21:41:38.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.766+0000 7f41174f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.766+0000 7f41174f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.766+0000 7f41174f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.766+0000 7f41174f1640 1 -- 172.21.15.120:0/3621380277 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411004ef90 con 0x7f411006ae80 2024-02-06T21:41:38.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2923fff640 1 -- 172.21.15.120:0/1062616246 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293400cba0 con 0x7f293c142680 2024-02-06T21:41:38.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2923fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2923fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:38.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2923fff640 10 monclient: dump: 2024-02-06T21:41:38.774 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.774 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.774 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.774 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.774 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.774 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.774 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.775 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.775 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.775 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2923fff640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2923fff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2923fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2923fff640 1 -- 172.21.15.120:0/1062616246 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293400cd00 con 0x7f293c142680 2024-02-06T21:41:38.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2923fff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2923fff640 1 -- 172.21.15.120:0/1062616246 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2934014680 con 0x7f293c142680 2024-02-06T21:41:38.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2923fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2923fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:38.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2923fff640 10 monclient: dump: 2024-02-06T21:41:38.775 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.775 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.776 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.776 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.776 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.776 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.776 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.776 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.776 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.776 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2941acf640 5 monclient: authenticate success, global_id 4301 2024-02-06T21:41:38.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2941acf640 10 monclient: _renew_subs 2024-02-06T21:41:38.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2941acf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.768+0000 7f2941acf640 1 -- 172.21.15.120:0/1062616246 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293c138340 con 0x7f293c142680 2024-02-06T21:41:38.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.769+0000 7f2941acf640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:38.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.769+0000 7f2941acf640 10 monclient: _renew_subs 2024-02-06T21:41:38.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.769+0000 7f2941acf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.769+0000 7f2941acf640 1 -- 172.21.15.120:0/1062616246 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293c136560 con 0x7f293c142680 2024-02-06T21:41:38.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.769+0000 7f2941acf640 1 librados: init done 2024-02-06T21:41:38.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.770+0000 7f2921ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.770+0000 7f2921ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:38.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.770+0000 7f2921ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:38.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.770+0000 7f2921ffb640 1 -- 172.21.15.120:0/1062616246 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291c005080 con 0x7f293c142680 2024-02-06T21:41:38.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.770+0000 7f2939ffb640 4 set_mon_vals no callback set 2024-02-06T21:41:38.808 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.806+0000 7f2923fff640 1 -- 172.21.15.120:0/1062616246 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f293401d3b0 con 0x7f293c142680 2024-02-06T21:41:38.808 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.807+0000 7f2923fff640 1 --2- 172.21.15.120:0/1062616246 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f292806d1f0 0x7f292806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.807+0000 7f293b7fe640 1 --2- 172.21.15.120:0/1062616246 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f292806d1f0 0x7f292806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.807+0000 7f293b7fe640 10 monclient: get_auth_request con 0x7f292806d1f0 auth_method 0 2024-02-06T21:41:38.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.808+0000 7f293b7fe640 1 --2- 172.21.15.120:0/1062616246 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f292806d1f0 0x7f292806f6b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f292c009870 tx=0x7f292c030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.808+0000 7f2923fff640 10 client.4301.objecter ms_handle_connect 0x7f292806d1f0 2024-02-06T21:41:38.921 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.919+0000 7f763b9c8640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-02-06T21:41:38.921 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.919+0000 7f763b9c8640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-02-06T21:41:38.921 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.919+0000 7f763b9c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:38.921 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.919+0000 7f763b9c8640 1 -- 172.21.15.120:0/784205979 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f763414fc90 con 0x7f7634157880 2024-02-06T21:41:38.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.934+0000 7f1f9e40f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-02-06T21:41:38.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.934+0000 7f1f9e40f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-02-06T21:41:38.936 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.934+0000 7f1f9e40f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:38.936 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.934+0000 7f1f9e40f640 1 -- 172.21.15.120:0/89737353 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f1f78005600 con 0x7f1f9806e7c0 2024-02-06T21:41:38.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10e3fff640 1 -- 172.21.15.120:0/2790733828 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc031870 con 0x7f10ec13ff40 2024-02-06T21:41:38.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10e3fff640 10 monclient: dump: 2024-02-06T21:41:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.991 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.991 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.991 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10e3fff640 10 monclient: _finish_auth 0 2024-02-06T21:41:38.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10e3fff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:38.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:38.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10e3fff640 1 -- 172.21.15.120:0/2790733828 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10dc0319d0 con 0x7f10ec13ff40 2024-02-06T21:41:38.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10e3fff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:38.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10f32d3640 5 monclient: authenticate success, global_id 4339 2024-02-06T21:41:38.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10e3fff640 1 -- 172.21.15.120:0/2790733828 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc031d60 con 0x7f10ec13ff40 2024-02-06T21:41:38.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:38.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:38.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10e3fff640 10 monclient: dump: 2024-02-06T21:41:38.993 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:38.993 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:38.993 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.993 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:38.993 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:38.993 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.993 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:38.993 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:38.993 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:38.993 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10f32d3640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:38.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7f10f32d3640 4 set_mon_vals no callback set 2024-02-06T21:41:38.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.986+0000 7fa1c6ffd640 1 -- 172.21.15.120:0/1429386313 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa1c0003990 con 0x7fa1d00b12c0 2024-02-06T21:41:38.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.987+0000 7fa1c6ffd640 1 --2- 172.21.15.120:0/1429386313 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fa1b406d1f0 0x7fa1b406f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:38.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.987+0000 7f7629ffb640 1 -- 172.21.15.120:0/784205979 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7630020020 con 0x7f7634157880 2024-02-06T21:41:38.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.987+0000 7f7629ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-02-06T21:41:38.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.987+0000 7f7629ffb640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:38.995 INFO:teuthology.orchestra.run.smithi120.stdout:73014444039 2024-02-06T21:41:38.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.987+0000 7f10f32d3640 10 monclient: shutdown 2024-02-06T21:41:38.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.987+0000 7fa1d6172640 1 --2- 172.21.15.120:0/1429386313 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fa1b406d1f0 0x7fa1b406f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:38.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.987+0000 7fa1d6172640 10 monclient: get_auth_request con 0x7fa1b406d1f0 auth_method 0 2024-02-06T21:41:38.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.987+0000 7f10f32d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:38.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.987+0000 7f10f32d3640 1 -- 172.21.15.120:0/2790733828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10ec13ff40 msgr2=0x7f10ec0a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:38.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.987+0000 7f10f32d3640 1 --2- 172.21.15.120:0/2790733828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10ec13ff40 0x7f10ec0a9860 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f10ec06deb0 tx=0x7f10dc02b910 comp rx=0 tx=0).stop 2024-02-06T21:41:38.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.987+0000 7f10f32d3640 1 -- 172.21.15.120:0/2790733828 shutdown_connections 2024-02-06T21:41:38.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.987+0000 7f10f32d3640 1 --2- 172.21.15.120:0/2790733828 >> v2:172.21.15.120:3300/0 conn(0x7f10ec142680 0x7f10ec142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.987+0000 7f10f32d3640 1 --2- 172.21.15.120:0/2790733828 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10ec13ff40 0x7f10ec0a9860 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:38.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.987+0000 7f10f32d3640 1 -- 172.21.15.120:0/2790733828 >> 172.21.15.120:0/2790733828 conn(0x7f10ec0b8660 msgr2=0x7f10ec0b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:38.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.988+0000 7f10f32d3640 1 -- 172.21.15.120:0/2790733828 shutdown_connections 2024-02-06T21:41:38.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.988+0000 7f10f32d3640 1 -- 172.21.15.120:0/2790733828 wait complete. 2024-02-06T21:41:38.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.988+0000 7fa1c6ffd640 10 client.4336.objecter ms_handle_connect 0x7fa1b406d1f0 2024-02-06T21:41:38.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.988+0000 7fa1d6172640 1 --2- 172.21.15.120:0/1429386313 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fa1b406d1f0 0x7fa1b406f6b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc036000 tx=0x7fa1cc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:38.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f10f32d3640 10 monclient: build_initial_monmap 2024-02-06T21:41:38.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f10f32d3640 10 monclient: monmap: 2024-02-06T21:41:38.998 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:38.998 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:38.998 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:38.998 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:38.998 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:38.998 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:38.998 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:38.999 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:38.999 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:38.999 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:38.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f10f32d3640 1 librados: starting msgr at 2024-02-06T21:41:38.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f10f32d3640 1 librados: starting objecter 2024-02-06T21:41:38.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f763b9c8640 10 librados: watch_flush enter 2024-02-06T21:41:38.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f763b9c8640 10 librados: watch_flush exit 2024-02-06T21:41:38.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f763b9c8640 20 client.4333.objecter shutdown clearing up homeless session... 2024-02-06T21:41:39.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f763b9c8640 10 client.4333.objecter successfully canceled tick 2024-02-06T21:41:39.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f10f32d3640 1 Processor -- start 2024-02-06T21:41:39.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f763b9c8640 1 -- 172.21.15.120:0/784205979 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f761806d220 msgr2=0x7f761806f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f763b9c8640 1 --2- 172.21.15.120:0/784205979 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f761806d220 0x7f761806f6e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f762c035000 tx=0x7f762c02e000 comp rx=0 tx=0).stop 2024-02-06T21:41:39.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f763b9c8640 10 monclient: shutdown 2024-02-06T21:41:39.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f10f32d3640 1 -- start start 2024-02-06T21:41:39.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f763b9c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:39.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f763b9c8640 1 -- 172.21.15.120:0/784205979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7634157880 msgr2=0x7f76340a9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f763b9c8640 1 --2- 172.21.15.120:0/784205979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7634157880 0x7f76340a9600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f763000c3c0 tx=0x7f763000c880 comp rx=0 tx=0).stop 2024-02-06T21:41:39.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f10f32d3640 1 librados: setting wanted keys 2024-02-06T21:41:39.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f10f32d3640 1 librados: calling monclient init 2024-02-06T21:41:39.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f10f32d3640 10 monclient: init 2024-02-06T21:41:39.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f763b9c8640 1 -- 172.21.15.120:0/784205979 shutdown_connections 2024-02-06T21:41:39.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f763b9c8640 1 --2- 172.21.15.120:0/784205979 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f761806d220 0x7f761806f6e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f763b9c8640 1 --2- 172.21.15.120:0/784205979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7634157880 0x7f76340a9600 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f763b9c8640 1 --2- 172.21.15.120:0/784205979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f763408b2a0 0x7f76340ab390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.989+0000 7f763b9c8640 1 --2- 172.21.15.120:0/784205979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7634070cd0 0x7f7634155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.990+0000 7f763b9c8640 1 -- 172.21.15.120:0/784205979 >> 172.21.15.120:0/784205979 conn(0x7f76340b0cc0 msgr2=0x7f763406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:39.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.990+0000 7f763b9c8640 1 -- 172.21.15.120:0/784205979 shutdown_connections 2024-02-06T21:41:39.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.990+0000 7f763b9c8640 1 -- 172.21.15.120:0/784205979 wait complete. 2024-02-06T21:41:39.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.990+0000 7f763b9c8640 1 librados: shutdown 2024-02-06T21:41:39.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.990+0000 7fa1c6ffd640 1 -- 172.21.15.120:0/1429386313 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa1c00587d0 con 0x7fa1d00b12c0 2024-02-06T21:41:39.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.990+0000 7fa1c6ffd640 10 client.4336.objecter ms_dispatch 0x7fa1d0000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:39.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.990+0000 7fa1c6ffd640 3 client.4336.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:39.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.990+0000 7fa1c6ffd640 3 client.4336.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:39.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.990+0000 7f10f32d3640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:39.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.990+0000 7fa1c6ffd640 20 client.4336.objecter dump_active .. 0 homeless 2024-02-06T21:41:39.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.990+0000 7f10f32d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-02-06T21:41:39.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.990+0000 7f10f32d3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10ec0b54b0 0x7f10ec1343a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:39.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.991+0000 7f10f32d3640 10 monclient(hunting): picked mon.noname-c con 0x7f10ec0b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:39.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.991+0000 7f10f32d3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10ec13ff40 0x7f10ec1362a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:39.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.991+0000 7f10f32d3640 10 monclient(hunting): picked mon.noname-a con 0x7f10ec13ff40 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:39.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.991+0000 7f10f32d3640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10ec142680 0x7f10ec1321f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:39.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.991+0000 7f10f32d3640 10 monclient(hunting): picked mon.noname-b con 0x7f10ec142680 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:39.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.991+0000 7f10f32d3640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:39.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.991+0000 7f10f32d3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec137ce0 con 0x7f10ec13ff40 2024-02-06T21:41:39.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.991+0000 7f10f32d3640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:39.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.991+0000 7f10f32d3640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec1367e0 con 0x7f10ec142680 2024-02-06T21:41:39.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.991+0000 7f10f32d3640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:39.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.991+0000 7f10f32d3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec0b79f0 con 0x7f10ec0b54b0 2024-02-06T21:41:39.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.991+0000 7f10f32d3640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:39.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:38.991+0000 7f10f32d3640 10 monclient(hunting): authenticate will time out at 1329.328982s 2024-02-06T21:41:39.008 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444039 for osd.4 2024-02-06T21:41:39.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.005+0000 7fa1c6ffd640 1 -- 172.21.15.120:0/1429386313 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa1c0054370 con 0x7fa1d00b12c0 2024-02-06T21:41:39.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.005+0000 7fa1c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:39.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.005+0000 7fa1c6ffd640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:39.009 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.008+0000 7f10f1870640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10ec142680 0x7f10ec1321f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:39.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.008+0000 7f10f1870640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10ec142680 0x7f10ec1321f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38736/0 (socket says 172.21.15.120:38736) 2024-02-06T21:41:39.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.008+0000 7f10f1870640 1 -- 172.21.15.120:0/4149880317 learned_addr learned my addr 172.21.15.120:0/4149880317 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:39.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.008+0000 7f10f1870640 10 monclient(hunting): get_auth_request con 0x7f10ec142680 auth_method 0 2024-02-06T21:41:39.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.008+0000 7f10f1870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:39.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.008+0000 7f10f1870640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:39.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.008+0000 7f10f1870640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:39.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.009+0000 7f10f1870640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:39.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.009+0000 7f10f1870640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:39.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.009+0000 7f10f1870640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:39.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.012+0000 7f10f1870640 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2024-02-06T21:41:39.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.012+0000 7f10f1870640 1 -- 172.21.15.120:0/4149880317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10ec0b54b0 msgr2=0x7f10ec1343a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.012+0000 7f10f1870640 1 --2- 172.21.15.120:0/4149880317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10ec0b54b0 0x7f10ec1343a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.012+0000 7f10f1870640 1 -- 172.21.15.120:0/4149880317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10ec13ff40 msgr2=0x7f10ec1362a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.012+0000 7f10f1870640 1 --2- 172.21.15.120:0/4149880317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10ec13ff40 0x7f10ec1362a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.012+0000 7f10f1870640 10 monclient: _finish_hunting 0 2024-02-06T21:41:39.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.012+0000 7f10f1870640 1 monclient: found mon.noname-b 2024-02-06T21:41:39.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.012+0000 7f10f1870640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:39.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.012+0000 7f10f1870640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:39.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.012+0000 7f10f1870640 1 -- 172.21.15.120:0/4149880317 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc0094d0 con 0x7f10ec142680 2024-02-06T21:41:39.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.014+0000 7f10f1870640 1 --2- 172.21.15.120:0/4149880317 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10ec142680 0x7f10ec1321f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f10e400b970 tx=0x7f10e400be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:39.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.014+0000 7f10e1ffb640 10 client.?.objecter ms_handle_connect 0x7f10ec142680 2024-02-06T21:41:39.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.014+0000 7f10e1ffb640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:39.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.014+0000 7f3ef67e6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-02-06T21:41:39.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.014+0000 7f3ef67e6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-02-06T21:41:39.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.014+0000 7f3ef67e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:39.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.014+0000 7f3ef67e6640 1 -- 172.21.15.120:0/1705276881 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3ef013c380 con 0x7f3ef006f920 2024-02-06T21:41:39.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10e1ffb640 1 -- 172.21.15.120:0/4149880317 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e400cb40 con 0x7f10ec142680 2024-02-06T21:41:39.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:39.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10e1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:39.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10e1ffb640 10 monclient: dump: 2024-02-06T21:41:39.021 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:39.021 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:39.021 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:39.021 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:39.021 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:39.021 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:39.021 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:39.022 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:39.022 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:39.022 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:39.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10e1ffb640 10 monclient: _finish_auth 0 2024-02-06T21:41:39.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10e1ffb640 10 monclient: _check_auth_tickets 2024-02-06T21:41:39.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:39.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10e1ffb640 1 -- 172.21.15.120:0/4149880317 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e400ccc0 con 0x7f10ec142680 2024-02-06T21:41:39.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10f32d3640 5 monclient: authenticate success, global_id 4342 2024-02-06T21:41:39.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:39.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10e1ffb640 1 -- 172.21.15.120:0/4149880317 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e4014680 con 0x7f10ec142680 2024-02-06T21:41:39.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10e37fe640 4 set_mon_vals no callback set 2024-02-06T21:41:39.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10f32d3640 10 monclient: _renew_subs 2024-02-06T21:41:39.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10f32d3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:39.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10f32d3640 1 -- 172.21.15.120:0/4149880317 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10ec0b74d0 con 0x7f10ec142680 2024-02-06T21:41:39.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:39.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:39.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10e1ffb640 10 monclient: dump: 2024-02-06T21:41:39.023 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:39.023 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:39.023 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:39.023 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:39.023 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:39.023 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:39.023 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:39.024 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:39.024 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:39.024 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:39.024 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10f32d3640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:39.024 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10f32d3640 10 monclient: _renew_subs 2024-02-06T21:41:39.024 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10f32d3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:39.024 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10f32d3640 1 -- 172.21.15.120:0/4149880317 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ec132730 con 0x7f10ec142680 2024-02-06T21:41:39.024 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.018+0000 7f10f32d3640 1 librados: init done 2024-02-06T21:41:39.024 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.019+0000 7f10f32d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:39.024 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.019+0000 7f10f32d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:39.024 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.019+0000 7f10f32d3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:39.024 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.019+0000 7f10f32d3640 1 -- 172.21.15.120:0/4149880317 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10ec04ef90 con 0x7f10ec142680 2024-02-06T21:41:39.030 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.028+0000 7ff8727fc640 1 -- 172.21.15.120:0/1040850199 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff874007070 con 0x7ff87c142680 2024-02-06T21:41:39.030 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.029+0000 7ff8727fc640 1 --2- 172.21.15.120:0/1040850199 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff86006d1a0 0x7ff86006f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:39.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.029+0000 7ff881a41640 1 --2- 172.21.15.120:0/1040850199 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff86006d1a0 0x7ff86006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:39.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.029+0000 7ff881a41640 10 monclient: get_auth_request con 0x7ff86006d1a0 auth_method 0 2024-02-06T21:41:39.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.029+0000 7ff881a41640 1 --2- 172.21.15.120:0/1040850199 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff86006d1a0 0x7ff86006f660 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff878009710 tx=0x7ff878006c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:39.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.029+0000 7ff8727fc640 10 client.4332.objecter ms_handle_connect 0x7ff86006d1a0 2024-02-06T21:41:39.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.035+0000 7f10e1ffb640 1 -- 172.21.15.120:0/4149880317 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f10e401d3b0 con 0x7f10ec142680 2024-02-06T21:41:39.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.036+0000 7f10e1ffb640 1 --2- 172.21.15.120:0/4149880317 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f10d006d1f0 0x7f10d006f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:39.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.036+0000 7f10f106f640 1 --2- 172.21.15.120:0/4149880317 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f10d006d1f0 0x7f10d006f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:39.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.036+0000 7f10f106f640 10 monclient: get_auth_request con 0x7f10d006d1f0 auth_method 0 2024-02-06T21:41:39.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.036+0000 7f10e1ffb640 10 client.4342.objecter ms_handle_connect 0x7f10d006d1f0 2024-02-06T21:41:39.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.036+0000 7f10f106f640 1 --2- 172.21.15.120:0/4149880317 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f10d006d1f0 0x7f10d006f6b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f10e8001560 tx=0x7f10e8007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:39.044 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.042+0000 7ff8727fc640 1 -- 172.21.15.120:0/1040850199 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff874058d30 con 0x7ff87c142680 2024-02-06T21:41:39.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.042+0000 7ff8727fc640 10 client.4332.objecter ms_dispatch 0x7ff87c000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:39.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.042+0000 7ff8727fc640 3 client.4332.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:39.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.042+0000 7ff8727fc640 3 client.4332.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:39.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.042+0000 7ff8727fc640 20 client.4332.objecter dump_active .. 0 homeless 2024-02-06T21:41:39.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.042+0000 7ff8727fc640 1 -- 172.21.15.120:0/1040850199 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff874054fd0 con 0x7ff87c142680 2024-02-06T21:41:39.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.042+0000 7ff8727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:39.045 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.042+0000 7ff8727fc640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:39.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.057+0000 7f1f94ff9640 1 -- 172.21.15.120:0/89737353 <== mon.0 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1f8c035070 con 0x7f1f9806e7c0 2024-02-06T21:41:39.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.057+0000 7f1f94ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-02-06T21:41:39.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.057+0000 7f1f94ff9640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:39.062 INFO:teuthology.orchestra.run.smithi120.stdout:55834574852 2024-02-06T21:41:39.064 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 10 librados: watch_flush enter 2024-02-06T21:41:39.064 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 10 librados: watch_flush exit 2024-02-06T21:41:39.064 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 20 client.4329.objecter shutdown clearing up homeless session... 2024-02-06T21:41:39.065 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 10 client.4329.objecter successfully canceled tick 2024-02-06T21:41:39.065 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 1 -- 172.21.15.120:0/89737353 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1f8406d1a0 msgr2=0x7f1f8406f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.065 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 1 --2- 172.21.15.120:0/89737353 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1f8406d1a0 0x7f1f8406f660 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1f980b22b0 tx=0x7f1f88007a40 comp rx=0 tx=0).stop 2024-02-06T21:41:39.065 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 10 monclient: shutdown 2024-02-06T21:41:39.065 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:39.065 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 1 -- 172.21.15.120:0/89737353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f9806e7c0 msgr2=0x7f1f98146410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.066 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 1 --2- 172.21.15.120:0/89737353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f9806e7c0 0x7f1f98146410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c000c00 tx=0x7f1f8c0046c0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.066 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 1 -- 172.21.15.120:0/89737353 shutdown_connections 2024-02-06T21:41:39.066 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 1 --2- 172.21.15.120:0/89737353 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1f8406d1a0 0x7f1f8406f660 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.066 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 1 --2- 172.21.15.120:0/89737353 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f9806f920 0x7f1f981417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.066 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 1 --2- 172.21.15.120:0/89737353 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1f98071110 0x7f1f98141cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.066 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 1 --2- 172.21.15.120:0/89737353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f9806e7c0 0x7f1f98146410 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.060+0000 7f1f9e40f640 1 -- 172.21.15.120:0/89737353 >> 172.21.15.120:0/89737353 conn(0x7f1f980b0bc0 msgr2=0x7f1f980744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:39.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.061+0000 7f1f9e40f640 1 -- 172.21.15.120:0/89737353 shutdown_connections 2024-02-06T21:41:39.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.061+0000 7f1f9e40f640 1 -- 172.21.15.120:0/89737353 wait complete. 2024-02-06T21:41:39.067 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.061+0000 7f1f9e40f640 1 librados: shutdown 2024-02-06T21:41:39.077 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2024-02-06T21:41:39.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.188+0000 7f2923fff640 1 -- 172.21.15.120:0/1062616246 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f29340594c0 con 0x7f293c142680 2024-02-06T21:41:39.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.188+0000 7f2923fff640 10 client.4301.objecter ms_dispatch 0x7f293c000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:39.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.188+0000 7f2923fff640 3 client.4301.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:39.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.188+0000 7f2923fff640 3 client.4301.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:39.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.188+0000 7f2923fff640 20 client.4301.objecter dump_active .. 0 homeless 2024-02-06T21:41:39.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.188+0000 7f2923fff640 1 -- 172.21.15.120:0/1062616246 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2934055060 con 0x7f293c142680 2024-02-06T21:41:39.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.189+0000 7f2923fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:39.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.189+0000 7f2923fff640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:39.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.205+0000 7f3eecff9640 1 -- 172.21.15.120:0/1705276881 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3ee0034070 con 0x7f3ef006f920 2024-02-06T21:41:39.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.205+0000 7f3eecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-02-06T21:41:39.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.205+0000 7f3eecff9640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:39.208 INFO:teuthology.orchestra.run.smithi120.stdout:77309411335 2024-02-06T21:41:39.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.206+0000 7f3ef67e6640 10 librados: watch_flush enter 2024-02-06T21:41:39.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.206+0000 7f3ef67e6640 10 librados: watch_flush exit 2024-02-06T21:41:39.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.206+0000 7f3ef67e6640 20 client.4292.objecter shutdown clearing up homeless session... 2024-02-06T21:41:39.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.206+0000 7f3ef67e6640 10 client.4292.objecter successfully canceled tick 2024-02-06T21:41:39.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.207+0000 7f3ef67e6640 1 -- 172.21.15.120:0/1705276881 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f3edc06d220 msgr2=0x7f3edc06f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.207+0000 7f3ef67e6640 1 --2- 172.21.15.120:0/1705276881 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f3edc06d220 0x7f3edc06f6e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3ee80070b0 tx=0x7f3ee8007040 comp rx=0 tx=0).stop 2024-02-06T21:41:39.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.207+0000 7f3ef67e6640 10 monclient: shutdown 2024-02-06T21:41:39.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.207+0000 7f3ef67e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:39.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.207+0000 7f3ef67e6640 1 -- 172.21.15.120:0/1705276881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ef006f920 msgr2=0x7f3ef013e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.207+0000 7f3ef67e6640 1 --2- 172.21.15.120:0/1705276881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ef006f920 0x7f3ef013e500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0000c00 tx=0x7f3ee0004730 comp rx=0 tx=0).stop 2024-02-06T21:41:39.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.207+0000 7f3ef67e6640 1 -- 172.21.15.120:0/1705276881 shutdown_connections 2024-02-06T21:41:39.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.207+0000 7f3ef67e6640 1 --2- 172.21.15.120:0/1705276881 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f3edc06d220 0x7f3edc06f6e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.207+0000 7f3ef67e6640 1 --2- 172.21.15.120:0/1705276881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ef0071110 0x7f3ef013ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.207+0000 7f3ef67e6640 1 --2- 172.21.15.120:0/1705276881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ef006f920 0x7f3ef013e500 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.207+0000 7f3ef67e6640 1 --2- 172.21.15.120:0/1705276881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ef006e7c0 0x7f3ef0149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.207+0000 7f3ef67e6640 1 -- 172.21.15.120:0/1705276881 >> 172.21.15.120:0/1705276881 conn(0x7f3ef00b0bc0 msgr2=0x7f3ef0143af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:39.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.207+0000 7f3ef67e6640 1 -- 172.21.15.120:0/1705276881 shutdown_connections 2024-02-06T21:41:39.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.207+0000 7f3ef67e6640 1 -- 172.21.15.120:0/1705276881 wait complete. 2024-02-06T21:41:39.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.207+0000 7f3ef67e6640 1 librados: shutdown 2024-02-06T21:41:39.219 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411336 got 77309411335 for osd.1 2024-02-06T21:41:39.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.234+0000 7ff883ca5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-02-06T21:41:39.236 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.234+0000 7ff883ca5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-02-06T21:41:39.236 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.234+0000 7ff883ca5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:39.236 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.234+0000 7ff883ca5640 1 -- 172.21.15.120:0/1040850199 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7ff87c04ef90 con 0x7ff87c142680 2024-02-06T21:41:39.263 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.261+0000 7f10e1ffb640 1 -- 172.21.15.120:0/4149880317 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f10e400ce30 con 0x7f10ec142680 2024-02-06T21:41:39.264 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.261+0000 7f10e1ffb640 10 client.4342.objecter ms_dispatch 0x7f10ec000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:39.264 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.261+0000 7f10e1ffb640 3 client.4342.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:39.264 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.261+0000 7f10e1ffb640 3 client.4342.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:39.264 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.262+0000 7f10e1ffb640 20 client.4342.objecter dump_active .. 0 homeless 2024-02-06T21:41:39.273 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.272+0000 7f10e1ffb640 1 -- 172.21.15.120:0/4149880317 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f10e4054fb0 con 0x7f10ec142680 2024-02-06T21:41:39.274 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.272+0000 7f10e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:39.274 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.272+0000 7f10e1ffb640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:39.328 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.326+0000 7fa1d83d6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-02-06T21:41:39.328 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.326+0000 7fa1d83d6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-02-06T21:41:39.328 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.326+0000 7fa1d83d6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:39.328 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.326+0000 7fa1d83d6640 1 -- 172.21.15.120:0/1429386313 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa1d00ae3e0 con 0x7fa1d00b12c0 2024-02-06T21:41:39.331 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.330+0000 7fa1c6ffd640 1 -- 172.21.15.120:0/1429386313 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa1c0017090 con 0x7fa1d00b12c0 2024-02-06T21:41:39.331 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.330+0000 7fa1c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-02-06T21:41:39.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.330+0000 7fa1c6ffd640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:39.332 INFO:teuthology.orchestra.run.smithi120.stdout:73014444039 2024-02-06T21:41:39.334 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 10 librados: watch_flush enter 2024-02-06T21:41:39.334 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 10 librados: watch_flush exit 2024-02-06T21:41:39.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 20 client.4336.objecter shutdown clearing up homeless session... 2024-02-06T21:41:39.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 10 client.4336.objecter successfully canceled tick 2024-02-06T21:41:39.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 1 -- 172.21.15.120:0/1429386313 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fa1b406d1f0 msgr2=0x7fa1b406f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 1 --2- 172.21.15.120:0/1429386313 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fa1b406d1f0 0x7fa1b406f6b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc036000 tx=0x7fa1cc008030 comp rx=0 tx=0).stop 2024-02-06T21:41:39.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 10 monclient: shutdown 2024-02-06T21:41:39.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:39.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 1 -- 172.21.15.120:0/1429386313 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1d00b12c0 msgr2=0x7fa1d00aac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 1 --2- 172.21.15.120:0/1429386313 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1d00b12c0 0x7fa1d00aac90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0001eb0 tx=0x7fa1c000c890 comp rx=0 tx=0).stop 2024-02-06T21:41:39.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 1 -- 172.21.15.120:0/1429386313 shutdown_connections 2024-02-06T21:41:39.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 1 --2- 172.21.15.120:0/1429386313 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fa1b406d1f0 0x7fa1b406f6b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 1 --2- 172.21.15.120:0/1429386313 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1d00b12c0 0x7fa1d00aac90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 1 --2- 172.21.15.120:0/1429386313 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1d013bfc0 0x7fa1d00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 1 --2- 172.21.15.120:0/1429386313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1d0071900 0x7fa1d00ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 1 -- 172.21.15.120:0/1429386313 >> 172.21.15.120:0/1429386313 conn(0x7fa1d00b07a0 msgr2=0x7fa1d00ae920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:39.336 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 1 -- 172.21.15.120:0/1429386313 shutdown_connections 2024-02-06T21:41:39.336 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 1 -- 172.21.15.120:0/1429386313 wait complete. 2024-02-06T21:41:39.336 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.332+0000 7fa1d83d6640 1 librados: shutdown 2024-02-06T21:41:39.345 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444039 for osd.3 2024-02-06T21:41:39.370 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.368+0000 7f4105ffb640 1 -- 172.21.15.120:0/3621380277 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f410c003b20 con 0x7f411006ae80 2024-02-06T21:41:39.370 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.369+0000 7f4105ffb640 1 --2- 172.21.15.120:0/3621380277 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f40f406d1a0 0x7f40f406f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:39.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.369+0000 7f4114a8c640 1 --2- 172.21.15.120:0/3621380277 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f40f406d1a0 0x7f40f406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:39.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.369+0000 7f4114a8c640 10 monclient: get_auth_request con 0x7f40f406d1a0 auth_method 0 2024-02-06T21:41:39.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.370+0000 7f4105ffb640 10 client.4335.objecter ms_handle_connect 0x7f40f406d1a0 2024-02-06T21:41:39.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.370+0000 7f4114a8c640 1 --2- 172.21.15.120:0/3621380277 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f40f406d1a0 0x7f40f406f660 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4100035000 tx=0x7f4100030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:39.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.388+0000 7f4105ffb640 1 -- 172.21.15.120:0/3621380277 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f410c0584e0 con 0x7f411006ae80 2024-02-06T21:41:39.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.388+0000 7f4105ffb640 10 client.4335.objecter ms_dispatch 0x7f4110000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:39.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.388+0000 7f4105ffb640 3 client.4335.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:39.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.388+0000 7f4105ffb640 3 client.4335.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:39.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.388+0000 7f4105ffb640 20 client.4335.objecter dump_active .. 0 homeless 2024-02-06T21:41:39.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.388+0000 7f4105ffb640 1 -- 172.21.15.120:0/3621380277 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f410c057bb0 con 0x7f411006ae80 2024-02-06T21:41:39.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.388+0000 7f4105ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:39.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.388+0000 7f4105ffb640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:39.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.447+0000 7f2921ffb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-02-06T21:41:39.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.447+0000 7f2921ffb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-02-06T21:41:39.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.447+0000 7f2921ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:39.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.447+0000 7f2921ffb640 1 -- 172.21.15.120:0/1062616246 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f291c005600 con 0x7f293c142680 2024-02-06T21:41:39.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.465+0000 7f2923fff640 1 -- 172.21.15.120:0/1062616246 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2934019030 con 0x7f293c142680 2024-02-06T21:41:39.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.465+0000 7f2923fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-02-06T21:41:39.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.465+0000 7f2923fff640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:39.467 INFO:teuthology.orchestra.run.smithi120.stdout:73014444039 2024-02-06T21:41:39.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 10 librados: watch_flush enter 2024-02-06T21:41:39.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 10 librados: watch_flush exit 2024-02-06T21:41:39.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 20 client.4301.objecter shutdown clearing up homeless session... 2024-02-06T21:41:39.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 10 client.4301.objecter successfully canceled tick 2024-02-06T21:41:39.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 1 -- 172.21.15.120:0/1062616246 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f292806d1f0 msgr2=0x7f292806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 1 --2- 172.21.15.120:0/1062616246 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f292806d1f0 0x7f292806f6b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f292c009870 tx=0x7f292c030000 comp rx=0 tx=0).stop 2024-02-06T21:41:39.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 10 monclient: shutdown 2024-02-06T21:41:39.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:39.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 1 -- 172.21.15.120:0/1062616246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c142680 msgr2=0x7f293c138ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 1 --2- 172.21.15.120:0/1062616246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c142680 0x7f293c138ca0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f293400b990 tx=0x7f293400be50 comp rx=0 tx=0).stop 2024-02-06T21:41:39.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 1 -- 172.21.15.120:0/1062616246 shutdown_connections 2024-02-06T21:41:39.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 1 --2- 172.21.15.120:0/1062616246 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f292806d1f0 0x7f292806f6b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.471 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 1 --2- 172.21.15.120:0/1062616246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f293c142680 0x7f293c138ca0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.471 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 1 --2- 172.21.15.120:0/1062616246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f293c13f830 0x7f293c138760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.471 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 1 --2- 172.21.15.120:0/1062616246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f293c0b54b0 0x7f293c135c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.471 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.467+0000 7f2921ffb640 1 -- 172.21.15.120:0/1062616246 >> 172.21.15.120:0/1062616246 conn(0x7f293c0b8660 msgr2=0x7f293c0744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:39.471 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.468+0000 7f2921ffb640 1 -- 172.21.15.120:0/1062616246 shutdown_connections 2024-02-06T21:41:39.471 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.468+0000 7f2921ffb640 1 -- 172.21.15.120:0/1062616246 wait complete. 2024-02-06T21:41:39.471 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.468+0000 7f2921ffb640 1 librados: shutdown 2024-02-06T21:41:39.481 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444039 for osd.2 2024-02-06T21:41:39.489 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.487+0000 7ff8727fc640 1 -- 172.21.15.120:0/1040850199 <== mon.0 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff874021070 con 0x7ff87c142680 2024-02-06T21:41:39.490 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.488+0000 7ff8727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-02-06T21:41:39.490 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.488+0000 7ff8727fc640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:39.490 INFO:teuthology.orchestra.run.smithi120.stdout:55834574853 2024-02-06T21:41:39.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 10 librados: watch_flush enter 2024-02-06T21:41:39.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 10 librados: watch_flush exit 2024-02-06T21:41:39.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 20 client.4332.objecter shutdown clearing up homeless session... 2024-02-06T21:41:39.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 10 client.4332.objecter successfully canceled tick 2024-02-06T21:41:39.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 1 -- 172.21.15.120:0/1040850199 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff86006d1a0 msgr2=0x7ff86006f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 1 --2- 172.21.15.120:0/1040850199 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff86006d1a0 0x7ff86006f660 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff878009710 tx=0x7ff878006c90 comp rx=0 tx=0).stop 2024-02-06T21:41:39.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 10 monclient: shutdown 2024-02-06T21:41:39.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:39.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 1 -- 172.21.15.120:0/1040850199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff87c142680 msgr2=0x7ff87c136c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 1 --2- 172.21.15.120:0/1040850199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff87c142680 0x7ff87c136c10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff87400c9f0 tx=0x7ff87400ceb0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 1 -- 172.21.15.120:0/1040850199 shutdown_connections 2024-02-06T21:41:39.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 1 --2- 172.21.15.120:0/1040850199 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff86006d1a0 0x7ff86006f660 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 1 --2- 172.21.15.120:0/1040850199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff87c13f830 0x7ff87c1366d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 1 --2- 172.21.15.120:0/1040850199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff87c142680 0x7ff87c136c10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 1 --2- 172.21.15.120:0/1040850199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff87c0b54b0 0x7ff87c139130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.493 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 1 -- 172.21.15.120:0/1040850199 >> 172.21.15.120:0/1040850199 conn(0x7ff87c0b8660 msgr2=0x7ff87c13a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:39.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 1 -- 172.21.15.120:0/1040850199 shutdown_connections 2024-02-06T21:41:39.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 1 -- 172.21.15.120:0/1040850199 wait complete. 2024-02-06T21:41:39.494 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.490+0000 7ff883ca5640 1 librados: shutdown 2024-02-06T21:41:39.503 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2024-02-06T21:41:39.555 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.553+0000 7f10f32d3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-02-06T21:41:39.555 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.553+0000 7f10f32d3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-02-06T21:41:39.555 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.553+0000 7f10f32d3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:39.555 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.553+0000 7f10f32d3640 1 -- 172.21.15.120:0/4149880317 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f10ec0b7bc0 con 0x7f10ec142680 2024-02-06T21:41:39.558 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.557+0000 7f10e1ffb640 1 -- 172.21.15.120:0/4149880317 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f10e4019090 con 0x7f10ec142680 2024-02-06T21:41:39.558 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.557+0000 7f10e1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-02-06T21:41:39.558 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.557+0000 7f10e1ffb640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:39.559 INFO:teuthology.orchestra.run.smithi120.stdout:73014444039 2024-02-06T21:41:39.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 10 librados: watch_flush enter 2024-02-06T21:41:39.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 10 librados: watch_flush exit 2024-02-06T21:41:39.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 20 client.4342.objecter shutdown clearing up homeless session... 2024-02-06T21:41:39.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 10 client.4342.objecter successfully canceled tick 2024-02-06T21:41:39.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 1 -- 172.21.15.120:0/4149880317 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f10d006d1f0 msgr2=0x7f10d006f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 1 --2- 172.21.15.120:0/4149880317 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f10d006d1f0 0x7f10d006f6b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f10e8001560 tx=0x7f10e8007990 comp rx=0 tx=0).stop 2024-02-06T21:41:39.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 10 monclient: shutdown 2024-02-06T21:41:39.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:39.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 1 -- 172.21.15.120:0/4149880317 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10ec142680 msgr2=0x7f10ec1321f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 1 --2- 172.21.15.120:0/4149880317 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10ec142680 0x7f10ec1321f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f10e400b970 tx=0x7f10e400be30 comp rx=0 tx=0).stop 2024-02-06T21:41:39.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 1 -- 172.21.15.120:0/4149880317 shutdown_connections 2024-02-06T21:41:39.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 1 --2- 172.21.15.120:0/4149880317 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f10d006d1f0 0x7f10d006f6b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 1 --2- 172.21.15.120:0/4149880317 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10ec142680 0x7f10ec1321f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 1 --2- 172.21.15.120:0/4149880317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10ec13ff40 0x7f10ec1362a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 1 --2- 172.21.15.120:0/4149880317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f10ec0b54b0 0x7f10ec1343a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 1 -- 172.21.15.120:0/4149880317 >> 172.21.15.120:0/4149880317 conn(0x7f10ec0b8660 msgr2=0x7f10ec13c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:39.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 1 -- 172.21.15.120:0/4149880317 shutdown_connections 2024-02-06T21:41:39.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 1 -- 172.21.15.120:0/4149880317 wait complete. 2024-02-06T21:41:39.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.559+0000 7f10f32d3640 1 librados: shutdown 2024-02-06T21:41:39.575 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444039 for osd.6 2024-02-06T21:41:39.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.644+0000 7f41174f1640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-02-06T21:41:39.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.644+0000 7f41174f1640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-02-06T21:41:39.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.644+0000 7f41174f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:39.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.644+0000 7f41174f1640 1 -- 172.21.15.120:0/3621380277 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f4110141950 con 0x7f411006ae80 2024-02-06T21:41:39.807 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.803+0000 7f4105ffb640 1 -- 172.21.15.120:0/3621380277 <== mon.0 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f410c017090 con 0x7f411006ae80 2024-02-06T21:41:39.807 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.803+0000 7f4105ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-02-06T21:41:39.807 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.803+0000 7f4105ffb640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:39.808 INFO:teuthology.orchestra.run.smithi120.stdout:73014444041 2024-02-06T21:41:39.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 10 librados: watch_flush enter 2024-02-06T21:41:39.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 10 librados: watch_flush exit 2024-02-06T21:41:39.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 20 client.4335.objecter shutdown clearing up homeless session... 2024-02-06T21:41:39.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 10 client.4335.objecter successfully canceled tick 2024-02-06T21:41:39.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 1 -- 172.21.15.120:0/3621380277 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f40f406d1a0 msgr2=0x7f40f406f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 1 --2- 172.21.15.120:0/3621380277 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f40f406d1a0 0x7f40f406f660 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4100035000 tx=0x7f4100030000 comp rx=0 tx=0).stop 2024-02-06T21:41:39.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 10 monclient: shutdown 2024-02-06T21:41:39.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:39.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 1 -- 172.21.15.120:0/3621380277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f411006ae80 msgr2=0x7f4110146f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:39.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 1 --2- 172.21.15.120:0/3621380277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f411006ae80 0x7f4110146f80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f410c00c440 tx=0x7f410c00c900 comp rx=0 tx=0).stop 2024-02-06T21:41:39.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 1 -- 172.21.15.120:0/3621380277 shutdown_connections 2024-02-06T21:41:39.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 1 --2- 172.21.15.120:0/3621380277 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f40f406d1a0 0x7f40f406f660 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 1 --2- 172.21.15.120:0/3621380277 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4110071530 0x7f411013f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 1 --2- 172.21.15.120:0/3621380277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f411006fdf0 0x7f411013eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 1 --2- 172.21.15.120:0/3621380277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f411006ae80 0x7f4110146f80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:39.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.805+0000 7f41174f1640 1 -- 172.21.15.120:0/3621380277 >> 172.21.15.120:0/3621380277 conn(0x7f41100b4f90 msgr2=0x7f411013c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:39.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.806+0000 7f41174f1640 1 -- 172.21.15.120:0/3621380277 shutdown_connections 2024-02-06T21:41:39.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.806+0000 7f41174f1640 1 -- 172.21.15.120:0/3621380277 wait complete. 2024-02-06T21:41:39.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:39.806+0000 7f41174f1640 1 librados: shutdown 2024-02-06T21:41:39.818 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444042 got 73014444041 for osd.5 2024-02-06T21:41:40.008 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-02-06T21:41:40.079 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-02-06T21:41:40.111 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.110+0000 7feaff025640 10 monclient: get_monmap_and_config 2024-02-06T21:41:40.111 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.110+0000 7feaff025640 10 monclient: build_initial_monmap 2024-02-06T21:41:40.111 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.110+0000 7feaff025640 10 monclient: monmap: 2024-02-06T21:41:40.112 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:40.112 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.112 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:40.111494+0000 2024-02-06T21:41:40.112 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:40.111494+0000 2024-02-06T21:41:40.112 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:40.112 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.112 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:40.112 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:40.112 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:40.112 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:40.112 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:40.112 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.110+0000 7feaff025640 1 Processor -- start 2024-02-06T21:41:40.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.111+0000 7feaff025640 1 -- start start 2024-02-06T21:41:40.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.111+0000 7feaff025640 10 monclient: init 2024-02-06T21:41:40.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.111+0000 7feaff025640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:40.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.111+0000 7feaff025640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-02-06T21:41:40.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.111+0000 7feaff025640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf80b54b0 0x7feaf80a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.111+0000 7feaff025640 10 monclient(hunting): picked mon.noname-b con 0x7feaf80b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:40.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.111+0000 7feaff025640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feaf813f830 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:40.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.111+0000 7feaff025640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feaf8142680 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:40.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.111+0000 7feaff025640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf8143200 con 0x7feaf8142680 2024-02-06T21:41:40.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.111+0000 7feaff025640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf80ab720 con 0x7feaf813f830 2024-02-06T21:41:40.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.111+0000 7feaff025640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.111+0000 7feaff025640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf80ab520 con 0x7feaf80b54b0 2024-02-06T21:41:40.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.111+0000 7feaff025640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:40.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.111+0000 7feaff025640 10 monclient(hunting): authenticate will time out at 1330.449151s 2024-02-06T21:41:40.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.112+0000 7feafcdc1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf80b54b0 0x7feaf80a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.115 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.112+0000 7feafd5c2640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7feaf8142680 0x7feaf8142af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:53074/0 (socket says 172.21.15.120:53074) 2024-02-06T21:41:40.116 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.112+0000 7feafd5c2640 1 -- 172.21.15.120:0/2506270650 learned_addr learned my addr 172.21.15.120:0/2506270650 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:41:40.116 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.112+0000 7feafcdc1640 10 monclient(hunting): get_auth_request con 0x7feaf80b54b0 auth_method 0 2024-02-06T21:41:40.116 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.112+0000 7feafcdc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.116 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.112+0000 7feafcdc1640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.116 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.112+0000 7feafcdc1640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.116 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.113+0000 7feafcdc1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.116 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.113+0000 7feafcdc1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.116 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.113+0000 7feafcdc1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.114+0000 7feafcdc1640 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2024-02-06T21:41:40.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.114+0000 7feafcdc1640 1 -- 172.21.15.120:0/2506270650 >> v1:172.21.15.135:6789/0 conn(0x7feaf813f830 legacy=0x7feaf813fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.114+0000 7feafcdc1640 1 -- 172.21.15.120:0/2506270650 >> v1:172.21.15.120:6789/0 conn(0x7feaf8142680 legacy=0x7feaf8142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.114+0000 7feafcdc1640 10 monclient: _finish_hunting 0 2024-02-06T21:41:40.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.114+0000 7feafcdc1640 1 monclient: found mon.noname-b 2024-02-06T21:41:40.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.114+0000 7feafcdc1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:40.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.114+0000 7feafcdc1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:40.117 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.114+0000 7feafcdc1640 1 -- 172.21.15.120:0/2506270650 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf80b4800 con 0x7feaf80b54b0 2024-02-06T21:41:40.118 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.117+0000 7feafcdc1640 1 --2- 172.21.15.120:0/2506270650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf80b54b0 0x7feaf80a9860 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feaec0097d0 tx=0x7feaec02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e989c3e50117631 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.122 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaf77fe640 1 -- 172.21.15.120:0/2506270650 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaec031890 con 0x7feaf80b54b0 2024-02-06T21:41:40.122 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaf77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.122 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaf77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:40.122 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaf77fe640 10 monclient: dump: 2024-02-06T21:41:40.122 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.122 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.122 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.122 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.122 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.122 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.123 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.123 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.123 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.123 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaf77fe640 10 monclient: _finish_auth 0 2024-02-06T21:41:40.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaf77fe640 10 monclient: _check_auth_tickets 2024-02-06T21:41:40.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaf77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:40.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaf77fe640 1 -- 172.21.15.120:0/2506270650 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaec0319f0 con 0x7feaf80b54b0 2024-02-06T21:41:40.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaf77fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:40.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaf77fe640 1 -- 172.21.15.120:0/2506270650 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaec031d80 con 0x7feaf80b54b0 2024-02-06T21:41:40.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaff025640 5 monclient: authenticate success, global_id 4304 2024-02-06T21:41:40.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaff025640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:40.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaff025640 4 set_mon_vals no callback set 2024-02-06T21:41:40.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaf77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaf77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:40.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.120+0000 7feaf77fe640 10 monclient: dump: 2024-02-06T21:41:40.124 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.124 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.124 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.124 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.124 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.124 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.124 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.124 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.124 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.124 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.121+0000 7feaff025640 10 monclient: shutdown 2024-02-06T21:41:40.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.121+0000 7feaff025640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:40.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.121+0000 7feaff025640 1 -- 172.21.15.120:0/2506270650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf80b54b0 msgr2=0x7feaf80a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.121+0000 7feaff025640 1 --2- 172.21.15.120:0/2506270650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf80b54b0 0x7feaf80a9860 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feaec0097d0 tx=0x7feaec02b930 comp rx=0 tx=0).stop 2024-02-06T21:41:40.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.121+0000 7feaff025640 1 -- 172.21.15.120:0/2506270650 shutdown_connections 2024-02-06T21:41:40.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.121+0000 7feaff025640 1 --2- 172.21.15.120:0/2506270650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf80b54b0 0x7feaf80a9860 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.121+0000 7feaff025640 1 -- 172.21.15.120:0/2506270650 >> 172.21.15.120:0/2506270650 conn(0x7feaf80b8660 msgr2=0x7feaf80b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:40.128 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.121+0000 7feaff025640 1 -- 172.21.15.120:0/2506270650 shutdown_connections 2024-02-06T21:41:40.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.121+0000 7feaff025640 1 -- 172.21.15.120:0/2506270650 wait complete. 2024-02-06T21:41:40.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.122+0000 7feaff025640 10 monclient: build_initial_monmap 2024-02-06T21:41:40.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.122+0000 7feaff025640 10 monclient: monmap: 2024-02-06T21:41:40.129 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:40.129 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:40.129 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:40.129 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:40.129 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:40.129 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.129 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:40.129 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:40.129 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:40.129 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.122+0000 7feaff025640 1 librados: starting msgr at 2024-02-06T21:41:40.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.122+0000 7feaff025640 1 librados: starting objecter 2024-02-06T21:41:40.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.122+0000 7feaff025640 1 Processor -- start 2024-02-06T21:41:40.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.122+0000 7feaff025640 1 -- start start 2024-02-06T21:41:40.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.122+0000 7feaff025640 1 librados: setting wanted keys 2024-02-06T21:41:40.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.122+0000 7feaff025640 1 librados: calling monclient init 2024-02-06T21:41:40.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.122+0000 7feaff025640 10 monclient: init 2024-02-06T21:41:40.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:40.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T21:41:40.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feaf80b54b0 0x7feaf8135750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 10 monclient(hunting): picked mon.noname-b con 0x7feaf80b54b0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:40.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf813f830 0x7feaf81324d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 10 monclient(hunting): picked mon.noname-c con 0x7feaf813f830 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:40.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaf8142680 0x7feaf8132a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 10 monclient(hunting): picked mon.noname-a con 0x7feaf8142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:40.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf8136a30 con 0x7feaf8142680 2024-02-06T21:41:40.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf8136d10 con 0x7feaf80b54b0 2024-02-06T21:41:40.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf8136ff0 con 0x7feaf813f830 2024-02-06T21:41:40.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:40.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaff025640 10 monclient(hunting): authenticate will time out at 1330.460945s 2024-02-06T21:41:40.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.123+0000 7feaf7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf813f830 0x7feaf81324d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feafcdc1640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feaf80b54b0 0x7feaf8135750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feafd5c2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaf8142680 0x7feaf8132a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feaf7fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf813f830 0x7feaf81324d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57070/0 (socket says 172.21.15.120:57070) 2024-02-06T21:41:40.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feaf7fff640 1 -- 172.21.15.120:0/907807584 learned_addr learned my addr 172.21.15.120:0/907807584 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:40.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feaf7fff640 10 monclient(hunting): get_auth_request con 0x7feaf813f830 auth_method 0 2024-02-06T21:41:40.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feaf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feaf7fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feaf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feafcdc1640 10 monclient(hunting): get_auth_request con 0x7feaf80b54b0 auth_method 0 2024-02-06T21:41:40.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feafcdc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feafcdc1640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feafcdc1640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feafd5c2640 10 monclient(hunting): get_auth_request con 0x7feaf8142680 auth_method 0 2024-02-06T21:41:40.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feafd5c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feafd5c2640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feafd5c2640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feaf7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feaf7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.124+0000 7feaf7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.125+0000 7feafcdc1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.125+0000 7feafcdc1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.125+0000 7feafcdc1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.125+0000 7feafd5c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.125+0000 7feafd5c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.125+0000 7feafd5c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.127+0000 7feaf7fff640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-02-06T21:41:40.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.127+0000 7feaf7fff640 1 -- 172.21.15.120:0/907807584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feaf80b54b0 msgr2=0x7feaf8135750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.127+0000 7feaf7fff640 1 --2- 172.21.15.120:0/907807584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feaf80b54b0 0x7feaf8135750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.127+0000 7feaf7fff640 1 -- 172.21.15.120:0/907807584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaf8142680 msgr2=0x7feaf8132a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.127+0000 7feaf7fff640 1 --2- 172.21.15.120:0/907807584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaf8142680 0x7feaf8132a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.127+0000 7feaf7fff640 10 monclient: _finish_hunting 0 2024-02-06T21:41:40.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.127+0000 7feaf7fff640 1 monclient: found mon.noname-c 2024-02-06T21:41:40.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.127+0000 7feaf7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:40.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.127+0000 7feaf7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:41:40.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.127+0000 7feaf7fff640 1 -- 172.21.15.120:0/907807584 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaec009400 con 0x7feaf813f830 2024-02-06T21:41:40.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.129+0000 7feaf7fff640 1 --2- 172.21.15.120:0/907807584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf813f830 0x7feaf81324d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7feae400c400 tx=0x7feae400c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.129+0000 7feaf57fa640 10 client.?.objecter ms_handle_connect 0x7feaf813f830 2024-02-06T21:41:40.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.129+0000 7feaf57fa640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:40.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaf57fa640 1 -- 172.21.15.120:0/907807584 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae40052d0 con 0x7feaf813f830 2024-02-06T21:41:40.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaf57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaf57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:40.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaf57fa640 10 monclient: dump: 2024-02-06T21:41:40.137 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.137 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.137 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.137 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.137 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.137 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.137 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.137 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.137 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.137 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaf57fa640 10 monclient: _finish_auth 0 2024-02-06T21:41:40.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaf57fa640 10 monclient: _check_auth_tickets 2024-02-06T21:41:40.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaf57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:40.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaf57fa640 1 -- 172.21.15.120:0/907807584 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae4005450 con 0x7feaf813f830 2024-02-06T21:41:40.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaf57fa640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:40.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaf57fa640 1 -- 172.21.15.120:0/907807584 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae4007260 con 0x7feaf813f830 2024-02-06T21:41:40.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaff025640 5 monclient: authenticate success, global_id 4307 2024-02-06T21:41:40.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaf57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaf57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:40.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaf57fa640 10 monclient: dump: 2024-02-06T21:41:40.138 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.138 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.138 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.138 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.138 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.138 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.138 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.138 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.139 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.139 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaff025640 10 monclient: _renew_subs 2024-02-06T21:41:40.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaff025640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:40.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaff025640 1 -- 172.21.15.120:0/907807584 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf8131fb0 con 0x7feaf813f830 2024-02-06T21:41:40.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.133+0000 7feaf6ffd640 4 set_mon_vals no callback set 2024-02-06T21:41:40.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.134+0000 7feaff025640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:40.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.134+0000 7feaff025640 10 monclient: _renew_subs 2024-02-06T21:41:40.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.134+0000 7feaff025640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:40.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.134+0000 7feaff025640 1 -- 172.21.15.120:0/907807584 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf8136630 con 0x7feaf813f830 2024-02-06T21:41:40.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.134+0000 7feaff025640 1 librados: init done 2024-02-06T21:41:40.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.135+0000 7feaff025640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.135+0000 7feaff025640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.135+0000 7feaff025640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:40.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.135+0000 7feaff025640 1 -- 172.21.15.120:0/907807584 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf81321a0 con 0x7feaf813f830 2024-02-06T21:41:40.156 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.154+0000 7feaf57fa640 1 -- 172.21.15.120:0/907807584 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7feae40073c0 con 0x7feaf813f830 2024-02-06T21:41:40.156 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.155+0000 7feaf57fa640 1 --2- 172.21.15.120:0/907807584 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7feadc06d1f0 0x7feadc06f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.156 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.155+0000 7feafcdc1640 1 --2- 172.21.15.120:0/907807584 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7feadc06d1f0 0x7feadc06f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.157 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.155+0000 7feafcdc1640 10 monclient: get_auth_request con 0x7feadc06d1f0 auth_method 0 2024-02-06T21:41:40.157 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.155+0000 7feafcdc1640 1 --2- 172.21.15.120:0/907807584 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7feadc06d1f0 0x7feadc06f6b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feaec036000 tx=0x7feaec03c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.157 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.155+0000 7feaf57fa640 10 client.4307.objecter ms_handle_connect 0x7feadc06d1f0 2024-02-06T21:41:40.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.184+0000 7f02bbe86640 10 monclient: get_monmap_and_config 2024-02-06T21:41:40.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.184+0000 7f02bbe86640 10 monclient: build_initial_monmap 2024-02-06T21:41:40.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.184+0000 7f02bbe86640 10 monclient: monmap: 2024-02-06T21:41:40.186 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:40.186 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.186 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:40.185649+0000 2024-02-06T21:41:40.186 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:40.185649+0000 2024-02-06T21:41:40.186 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:40.186 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.186 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:40.187 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:40.187 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:40.187 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:40.187 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:40.187 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.184+0000 7f02bbe86640 1 Processor -- start 2024-02-06T21:41:40.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 1 -- start start 2024-02-06T21:41:40.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 10 monclient: init 2024-02-06T21:41:40.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:40.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-02-06T21:41:40.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b40b54b0 0x7f02b40a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 10 monclient(hunting): picked mon.noname-b con 0x7f02b40b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:40.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f02b413f830 0x7f02b413fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 10 monclient(hunting): picked mon.noname-a con 0x7f02b413f830 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:40.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f02b4142680 0x7f02b4142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 10 monclient(hunting): picked mon.noname-c con 0x7f02b4142680 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:40.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b40ab520 con 0x7f02b413f830 2024-02-06T21:41:40.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b40b5890 con 0x7f02b4142680 2024-02-06T21:41:40.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b40aa450 con 0x7f02b40b54b0 2024-02-06T21:41:40.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:40.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.185+0000 7f02bbe86640 10 monclient(hunting): authenticate will time out at 1330.523245s 2024-02-06T21:41:40.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.186+0000 7f02b9421640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f02b413f830 0x7f02b413fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.186+0000 7f02ba423640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f02b4142680 0x7f02b4142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.186+0000 7f02b9421640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f02b413f830 0x7f02b413fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60316/0 (socket says 172.21.15.120:60316) 2024-02-06T21:41:40.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.186+0000 7f02b9421640 1 -- 172.21.15.120:0/2650070701 learned_addr learned my addr 172.21.15.120:0/2650070701 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:40.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.186+0000 7f02b9421640 10 monclient(hunting): get_auth_request con 0x7f02b413f830 auth_method 0 2024-02-06T21:41:40.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.186+0000 7f02b9421640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.186+0000 7f02b9421640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.186+0000 7f02b9421640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.186+0000 7f02ba423640 10 monclient(hunting): get_auth_request con 0x7f02b4142680 auth_method 0 2024-02-06T21:41:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.186+0000 7f02ba423640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.186+0000 7f02ba423640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.186+0000 7f02ba423640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.187+0000 7f02b9421640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.187+0000 7f02b9421640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.187+0000 7f02b9421640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.187+0000 7f02ba423640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.187+0000 7f02ba423640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.187+0000 7f02ba423640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.189+0000 7f02b9421640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2024-02-06T21:41:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.189+0000 7f02b9421640 1 -- 172.21.15.120:0/2650070701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b40b54b0 msgr2=0x7f02b40a9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.189+0000 7f02b9421640 1 --2- 172.21.15.120:0/2650070701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b40b54b0 0x7f02b40a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.189+0000 7f02b9421640 1 -- 172.21.15.120:0/2650070701 >> v2:172.21.15.135:3300/0 conn(0x7f02b4142680 msgr2=0x7f02b4142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.189+0000 7f02b9421640 1 --2- 172.21.15.120:0/2650070701 >> v2:172.21.15.135:3300/0 conn(0x7f02b4142680 0x7f02b4142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.189+0000 7f02b9421640 10 monclient: _finish_hunting 0 2024-02-06T21:41:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.189+0000 7f02b9421640 1 monclient: found mon.noname-a 2024-02-06T21:41:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.189+0000 7f02b9421640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.189+0000 7f02b9421640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.189+0000 7f02b9421640 1 -- 172.21.15.120:0/2650070701 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b40ab720 con 0x7f02b413f830 2024-02-06T21:41:40.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.191+0000 7f02b9421640 1 --2- 172.21.15.120:0/2650070701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b413f830 0x7f02b413fc70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f02b4073210 tx=0x7f02a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974452632feb21f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02b8c20640 1 -- 172.21.15.120:0/2650070701 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4031640 con 0x7f02b413f830 2024-02-06T21:41:40.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02b8c20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02b8c20640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:40.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02b8c20640 10 monclient: dump: 2024-02-06T21:41:40.198 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.198 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.198 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.198 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.198 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.198 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.199 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.199 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.199 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.199 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02b8c20640 10 monclient: _finish_auth 0 2024-02-06T21:41:40.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02b8c20640 10 monclient: _check_auth_tickets 2024-02-06T21:41:40.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02b8c20640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:40.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02b8c20640 1 -- 172.21.15.120:0/2650070701 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a40317a0 con 0x7f02b413f830 2024-02-06T21:41:40.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02b8c20640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:40.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02b8c20640 1 -- 172.21.15.120:0/2650070701 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4031b30 con 0x7f02b413f830 2024-02-06T21:41:40.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02bbe86640 5 monclient: authenticate success, global_id 4347 2024-02-06T21:41:40.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02bbe86640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:40.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02bbe86640 4 set_mon_vals no callback set 2024-02-06T21:41:40.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02b8c20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02b8c20640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:40.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.195+0000 7f02b8c20640 10 monclient: dump: 2024-02-06T21:41:40.200 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.200 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.200 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.200 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.200 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.200 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.200 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.200 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.200 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.200 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.196+0000 7f02bbe86640 10 monclient: shutdown 2024-02-06T21:41:40.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.196+0000 7f02bbe86640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:40.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.196+0000 7f02bbe86640 1 -- 172.21.15.120:0/2650070701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b413f830 msgr2=0x7f02b413fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.196+0000 7f02bbe86640 1 --2- 172.21.15.120:0/2650070701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b413f830 0x7f02b413fc70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f02b4073210 tx=0x7f02a402b8f0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.196+0000 7f02bbe86640 1 -- 172.21.15.120:0/2650070701 shutdown_connections 2024-02-06T21:41:40.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.196+0000 7f02bbe86640 1 --2- 172.21.15.120:0/2650070701 >> v2:172.21.15.135:3300/0 conn(0x7f02b4142680 0x7f02b4142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.196+0000 7f02bbe86640 1 --2- 172.21.15.120:0/2650070701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b413f830 0x7f02b413fc70 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.196+0000 7f02bbe86640 1 --2- 172.21.15.120:0/2650070701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b40b54b0 0x7f02b40a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.196+0000 7f02bbe86640 1 -- 172.21.15.120:0/2650070701 >> 172.21.15.120:0/2650070701 conn(0x7f02b40b8660 msgr2=0x7f02b40b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:40.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.196+0000 7f02bbe86640 1 -- 172.21.15.120:0/2650070701 shutdown_connections 2024-02-06T21:41:40.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.196+0000 7f02bbe86640 1 -- 172.21.15.120:0/2650070701 wait complete. 2024-02-06T21:41:40.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.196+0000 7f02bbe86640 10 monclient: build_initial_monmap 2024-02-06T21:41:40.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.196+0000 7f02bbe86640 10 monclient: monmap: 2024-02-06T21:41:40.201 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:40.201 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:40.201 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:40.201 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:40.202 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:40.202 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.202 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:40.202 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:40.202 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:40.202 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 1 librados: starting msgr at 2024-02-06T21:41:40.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 1 librados: starting objecter 2024-02-06T21:41:40.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 1 Processor -- start 2024-02-06T21:41:40.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 1 -- start start 2024-02-06T21:41:40.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 1 librados: setting wanted keys 2024-02-06T21:41:40.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 1 librados: calling monclient init 2024-02-06T21:41:40.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 10 monclient: init 2024-02-06T21:41:40.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:40.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 10 monclient: _add_conns ranks=[2,0,1] 2024-02-06T21:41:40.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b40b54b0 0x7f02b41d82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 10 monclient(hunting): picked mon.noname-c con 0x7f02b40b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:40.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b413f830 0x7f02b40b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 10 monclient(hunting): picked mon.noname-a con 0x7f02b413f830 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:40.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02b4142680 0x7f02b41d76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 10 monclient(hunting): picked mon.noname-b con 0x7f02b4142680 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:40.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b41d5150 con 0x7f02b413f830 2024-02-06T21:41:40.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b40b0800 con 0x7f02b4142680 2024-02-06T21:41:40.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b40b0a30 con 0x7f02b40b54b0 2024-02-06T21:41:40.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:40.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.197+0000 7f02bbe86640 10 monclient(hunting): authenticate will time out at 1330.535410s 2024-02-06T21:41:40.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.198+0000 7f02b9421640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b413f830 0x7f02b40b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.198+0000 7f02b9421640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b413f830 0x7f02b40b1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60324/0 (socket says 172.21.15.120:60324) 2024-02-06T21:41:40.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.198+0000 7f02b9421640 1 -- 172.21.15.120:0/3440068232 learned_addr learned my addr 172.21.15.120:0/3440068232 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:40.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.198+0000 7f02b9421640 10 monclient(hunting): get_auth_request con 0x7f02b413f830 auth_method 0 2024-02-06T21:41:40.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.198+0000 7f02b9421640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.198+0000 7f02b9421640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.198+0000 7f02b9421640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.199+0000 7f02b9421640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.199+0000 7f02b9421640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.199+0000 7f02b9421640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.202+0000 7f02b9421640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-02-06T21:41:40.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.202+0000 7f02b9421640 1 -- 172.21.15.120:0/3440068232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b40b54b0 msgr2=0x7f02b41d82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.202+0000 7f02b9421640 1 --2- 172.21.15.120:0/3440068232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b40b54b0 0x7f02b41d82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.202+0000 7f02b9421640 1 -- 172.21.15.120:0/3440068232 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02b4142680 msgr2=0x7f02b41d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.202+0000 7f02b9421640 1 --2- 172.21.15.120:0/3440068232 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02b4142680 0x7f02b41d76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.202+0000 7f02b9421640 10 monclient: _finish_hunting 0 2024-02-06T21:41:40.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.202+0000 7f02b9421640 1 monclient: found mon.noname-a 2024-02-06T21:41:40.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.202+0000 7f02b9421640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:40.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.202+0000 7f02b9421640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:40.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.202+0000 7f02b9421640 1 -- 172.21.15.120:0/3440068232 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a40094d0 con 0x7f02b413f830 2024-02-06T21:41:40.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.204+0000 7f02b9421640 1 --2- 172.21.15.120:0/3440068232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b413f830 0x7f02b40b1190 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f02a40097f0 tx=0x7f02a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.205+0000 7f02aa7fc640 10 client.?.objecter ms_handle_connect 0x7f02b413f830 2024-02-06T21:41:40.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.205+0000 7f02aa7fc640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:40.211 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02aa7fc640 1 -- 172.21.15.120:0/3440068232 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a403a8b0 con 0x7f02b413f830 2024-02-06T21:41:40.211 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.211 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02aa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:40.211 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02aa7fc640 10 monclient: dump: 2024-02-06T21:41:40.211 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.211 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.211 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.212 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.212 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.212 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.212 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.212 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.212 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.212 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02aa7fc640 10 monclient: _finish_auth 0 2024-02-06T21:41:40.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02aa7fc640 10 monclient: _check_auth_tickets 2024-02-06T21:41:40.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:40.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02aa7fc640 1 -- 172.21.15.120:0/3440068232 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a403aa10 con 0x7f02b413f830 2024-02-06T21:41:40.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:40.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02aa7fc640 1 -- 172.21.15.120:0/3440068232 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a403adc0 con 0x7f02b413f830 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02bbe86640 5 monclient: authenticate success, global_id 4350 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02bbe86640 10 monclient: _renew_subs 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02bbe86640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02bbe86640 1 -- 172.21.15.120:0/3440068232 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b41d7e20 con 0x7f02b413f830 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02abfff640 4 set_mon_vals no callback set 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02aa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02aa7fc640 10 monclient: dump: 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.213 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.214 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.214 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02bbe86640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:40.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02bbe86640 10 monclient: _renew_subs 2024-02-06T21:41:40.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02bbe86640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:40.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02bbe86640 1 -- 172.21.15.120:0/3440068232 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b40ae330 con 0x7f02b413f830 2024-02-06T21:41:40.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.209+0000 7f02bbe86640 1 librados: init done 2024-02-06T21:41:40.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.210+0000 7f02bbe86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.210+0000 7f02bbe86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.210+0000 7f02bbe86640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:40.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.210+0000 7f02bbe86640 1 -- 172.21.15.120:0/3440068232 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b41d8000 con 0x7f02b413f830 2024-02-06T21:41:40.221 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-02-06T21:41:40.322 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.320+0000 7fc59cf82640 10 monclient: get_monmap_and_config 2024-02-06T21:41:40.322 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.320+0000 7fc59cf82640 10 monclient: build_initial_monmap 2024-02-06T21:41:40.323 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.320+0000 7fc59cf82640 10 monclient: monmap: 2024-02-06T21:41:40.323 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:40.323 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.323 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:40.322024+0000 2024-02-06T21:41:40.323 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:40.322024+0000 2024-02-06T21:41:40.323 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:40.323 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.323 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:40.323 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:40.323 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:40.323 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:40.323 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:40.324 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.324 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.321+0000 7fc59cf82640 1 Processor -- start 2024-02-06T21:41:40.324 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.321+0000 7fc59cf82640 1 -- start start 2024-02-06T21:41:40.324 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.321+0000 7fc59cf82640 10 monclient: init 2024-02-06T21:41:40.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.322+0000 7fc59cf82640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:40.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.322+0000 7fc59cf82640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-02-06T21:41:40.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.322+0000 7fc59cf82640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980a9c80 0x7fc5980aa060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.322+0000 7fc59cf82640 10 monclient(hunting): picked mon.noname-b con 0x7fc5980a9c80 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:40.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.322+0000 7fc59cf82640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc598142bb0 0x7fc598142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.322+0000 7fc59cf82640 10 monclient(hunting): picked mon.noname-a con 0x7fc598142bb0 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:40.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.322+0000 7fc59cf82640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc59813f810 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:40.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.322+0000 7fc59cf82640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5980aa5a0 con 0x7fc59813f810 2024-02-06T21:41:40.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.322+0000 7fc59cf82640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.322+0000 7fc59cf82640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc598143510 con 0x7fc598142bb0 2024-02-06T21:41:40.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.322+0000 7fc59cf82640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.322+0000 7fc59cf82640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc598142720 con 0x7fc5980a9c80 2024-02-06T21:41:40.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.322+0000 7fc59cf82640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:40.326 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.322+0000 7fc59cf82640 10 monclient(hunting): authenticate will time out at 1330.659636s 2024-02-06T21:41:40.346 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-02-06T21:41:40.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.405+0000 7f02aa7fc640 1 -- 172.21.15.120:0/3440068232 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f02a402f050 con 0x7f02b413f830 2024-02-06T21:41:40.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.406+0000 7f02aa7fc640 1 --2- 172.21.15.120:0/3440068232 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f029806d1f0 0x7f029806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.406+0000 7f02b9c22640 1 --2- 172.21.15.120:0/3440068232 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f029806d1f0 0x7f029806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.406+0000 7f02b9c22640 10 monclient: get_auth_request con 0x7f029806d1f0 auth_method 0 2024-02-06T21:41:40.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.407+0000 7f02aa7fc640 10 client.4350.objecter ms_handle_connect 0x7f029806d1f0 2024-02-06T21:41:40.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.407+0000 7f02b9c22640 1 --2- 172.21.15.120:0/3440068232 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f029806d1f0 0x7f029806f6b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009670 tx=0x7f02b0006bf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.411+0000 7f02aa7fc640 1 -- 172.21.15.120:0/3440068232 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f02a407b2f0 con 0x7f02b413f830 2024-02-06T21:41:40.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.411+0000 7f02aa7fc640 10 client.4350.objecter ms_dispatch 0x7f02b4000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:40.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.411+0000 7f02aa7fc640 3 client.4350.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:40.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.411+0000 7f02aa7fc640 3 client.4350.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:40.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.411+0000 7f02aa7fc640 20 client.4350.objecter dump_active .. 0 homeless 2024-02-06T21:41:40.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.416+0000 7feaf57fa640 1 -- 172.21.15.120:0/907807584 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7feae4007670 con 0x7feaf813f830 2024-02-06T21:41:40.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.416+0000 7feaf57fa640 10 client.4307.objecter ms_dispatch 0x7feaf8000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:40.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.416+0000 7feaf57fa640 3 client.4307.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:40.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.416+0000 7feaf57fa640 3 client.4307.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:40.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.416+0000 7feaf57fa640 20 client.4307.objecter dump_active .. 0 homeless 2024-02-06T21:41:40.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.416+0000 7feaf57fa640 1 -- 172.21.15.120:0/907807584 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7feae4054880 con 0x7feaf813f830 2024-02-06T21:41:40.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.416+0000 7feaf57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.416+0000 7feaf57fa640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:40.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.421+0000 7fc59659c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980a9c80 0x7fc5980aa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.422+0000 7fc59659c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980a9c80 0x7fc5980aa060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57090/0 (socket says 172.21.15.120:57090) 2024-02-06T21:41:40.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.422+0000 7fc59659c640 1 -- 172.21.15.120:0/1061220982 learned_addr learned my addr 172.21.15.120:0/1061220982 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:40.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.422+0000 7fc59659c640 10 monclient(hunting): get_auth_request con 0x7fc5980a9c80 auth_method 0 2024-02-06T21:41:40.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.422+0000 7fc59659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.422+0000 7fc59659c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.422+0000 7fc59659c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.423+0000 7fc59659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.423+0000 7fc59659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.423+0000 7fc59659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.427+0000 7fc59659c640 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2024-02-06T21:41:40.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.427+0000 7fc59659c640 1 -- 172.21.15.120:0/1061220982 >> v2:172.21.15.120:3300/0 conn(0x7fc598142bb0 msgr2=0x7fc598142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:40.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.427+0000 7fc59659c640 1 --2- 172.21.15.120:0/1061220982 >> v2:172.21.15.120:3300/0 conn(0x7fc598142bb0 0x7fc598142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.427+0000 7fc59659c640 1 -- 172.21.15.120:0/1061220982 >> v1:172.21.15.120:6789/0 conn(0x7fc59813f810 legacy=0x7fc59813fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:40.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.427+0000 7fc59659c640 10 monclient: _finish_hunting 0 2024-02-06T21:41:40.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.427+0000 7fc59659c640 1 monclient: found mon.noname-b 2024-02-06T21:41:40.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.427+0000 7fc59659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:40.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.427+0000 7fc59659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:40.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.427+0000 7fc59659c640 1 -- 172.21.15.120:0/1061220982 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5980ab120 con 0x7fc5980a9c80 2024-02-06T21:41:40.433 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.432+0000 7fc59659c640 1 --2- 172.21.15.120:0/1061220982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980a9c80 0x7fc5980aa060 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc588009870 tx=0x7fc58802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcd7219aac65dd4b server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59559a640 1 -- 172.21.15.120:0/1061220982 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc588031640 con 0x7fc5980a9c80 2024-02-06T21:41:40.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:40.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59559a640 10 monclient: dump: 2024-02-06T21:41:40.440 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.440 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.440 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.440 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.441 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.441 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.441 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.441 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.441 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.441 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59559a640 10 monclient: _finish_auth 0 2024-02-06T21:41:40.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59559a640 10 monclient: _check_auth_tickets 2024-02-06T21:41:40.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:40.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59559a640 1 -- 172.21.15.120:0/1061220982 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5880317a0 con 0x7fc5980a9c80 2024-02-06T21:41:40.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59cf82640 5 monclient: authenticate success, global_id 4310 2024-02-06T21:41:40.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59cf82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:40.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59559a640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:40.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59cf82640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:40.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59cf82640 4 set_mon_vals no callback set 2024-02-06T21:41:40.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59559a640 1 -- 172.21.15.120:0/1061220982 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc588031b30 con 0x7fc5980a9c80 2024-02-06T21:41:40.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:40.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.438+0000 7fc59559a640 10 monclient: dump: 2024-02-06T21:41:40.442 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.442 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.442 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.442 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.442 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.442 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.442 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.443 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.443 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.443 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.439+0000 7fc59cf82640 10 monclient: shutdown 2024-02-06T21:41:40.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.439+0000 7fc59cf82640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:40.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.439+0000 7fc59cf82640 1 -- 172.21.15.120:0/1061220982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980a9c80 msgr2=0x7fc5980aa060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.439+0000 7fc59cf82640 1 --2- 172.21.15.120:0/1061220982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980a9c80 0x7fc5980aa060 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc588009870 tx=0x7fc58802b8f0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.439+0000 7fc59cf82640 1 -- 172.21.15.120:0/1061220982 shutdown_connections 2024-02-06T21:41:40.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.439+0000 7fc59cf82640 1 --2- 172.21.15.120:0/1061220982 >> v2:172.21.15.120:3300/0 conn(0x7fc598142bb0 0x7fc598142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.439+0000 7fc59cf82640 1 --2- 172.21.15.120:0/1061220982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980a9c80 0x7fc5980aa060 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.439+0000 7fc59cf82640 1 -- 172.21.15.120:0/1061220982 >> 172.21.15.120:0/1061220982 conn(0x7fc5980b4ba0 msgr2=0x7fc5980b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:40.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.439+0000 7fc59cf82640 1 -- 172.21.15.120:0/1061220982 shutdown_connections 2024-02-06T21:41:40.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.439+0000 7fc59cf82640 1 -- 172.21.15.120:0/1061220982 wait complete. 2024-02-06T21:41:40.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.440+0000 7fc59cf82640 10 monclient: build_initial_monmap 2024-02-06T21:41:40.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.440+0000 7fc59cf82640 10 monclient: monmap: 2024-02-06T21:41:40.447 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:40.447 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:40.447 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:40.447 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:40.447 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:40.447 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.447 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:40.447 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:40.447 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:40.447 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.441+0000 7fc59cf82640 1 librados: starting msgr at 2024-02-06T21:41:40.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.441+0000 7fc59cf82640 1 librados: starting objecter 2024-02-06T21:41:40.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.441+0000 7fc59cf82640 1 Processor -- start 2024-02-06T21:41:40.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.441+0000 7fc59cf82640 1 -- start start 2024-02-06T21:41:40.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.441+0000 7fc59cf82640 1 librados: setting wanted keys 2024-02-06T21:41:40.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.441+0000 7fc59cf82640 1 librados: calling monclient init 2024-02-06T21:41:40.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.441+0000 7fc59cf82640 10 monclient: init 2024-02-06T21:41:40.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:40.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T21:41:40.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980a9c80 0x7fc59813a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 10 monclient(hunting): picked mon.noname-c con 0x7fc5980a9c80 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:40.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc59813f810 0x7fc598136730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 10 monclient(hunting): picked mon.noname-b con 0x7fc59813f810 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:40.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc598142bb0 0x7fc5981354e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 10 monclient(hunting): picked mon.noname-a con 0x7fc598142bb0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:40.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5981350e0 con 0x7fc598142bb0 2024-02-06T21:41:40.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc598136c70 con 0x7fc59813f810 2024-02-06T21:41:40.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc598143e50 con 0x7fc5980a9c80 2024-02-06T21:41:40.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:40.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.442+0000 7fc59cf82640 10 monclient(hunting): authenticate will time out at 1330.780102s 2024-02-06T21:41:40.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.443+0000 7fc59659c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980a9c80 0x7fc59813a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.443+0000 7fc595d9b640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc59813f810 0x7fc598136730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.443+0000 7fc59659c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980a9c80 0x7fc59813a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57092/0 (socket says 172.21.15.120:57092) 2024-02-06T21:41:40.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.443+0000 7fc595d9b640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc59813f810 0x7fc598136730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38766/0 (socket says 172.21.15.120:38766) 2024-02-06T21:41:40.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.443+0000 7fc59659c640 1 -- 172.21.15.120:0/1578405434 learned_addr learned my addr 172.21.15.120:0/1578405434 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:40.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.443+0000 7fc59659c640 10 monclient(hunting): get_auth_request con 0x7fc5980a9c80 auth_method 0 2024-02-06T21:41:40.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.443+0000 7fc59659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.443+0000 7fc59659c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.443+0000 7fc59659c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.443+0000 7fc595d9b640 10 monclient(hunting): get_auth_request con 0x7fc59813f810 auth_method 0 2024-02-06T21:41:40.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.443+0000 7fc595d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.443+0000 7fc595d9b640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.443+0000 7fc595d9b640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.444+0000 7fc595d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.444+0000 7fc595d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.444+0000 7fc595d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.444+0000 7fc59659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.444+0000 7fc59659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.444+0000 7fc59659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.451 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.447+0000 7fc595d9b640 10 monclient(hunting): handle_auth_done global_id 4354 payload 824 2024-02-06T21:41:40.451 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.447+0000 7fc595d9b640 1 -- 172.21.15.120:0/1578405434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980a9c80 msgr2=0x7fc59813a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.451 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.447+0000 7fc595d9b640 1 --2- 172.21.15.120:0/1578405434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980a9c80 0x7fc59813a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.451 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.447+0000 7fc595d9b640 1 -- 172.21.15.120:0/1578405434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc598142bb0 msgr2=0x7fc5981354e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.447+0000 7fc595d9b640 1 --2- 172.21.15.120:0/1578405434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc598142bb0 0x7fc5981354e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.447+0000 7fc595d9b640 10 monclient: _finish_hunting 0 2024-02-06T21:41:40.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.447+0000 7fc595d9b640 1 monclient: found mon.noname-b 2024-02-06T21:41:40.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.447+0000 7fc595d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:40.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.447+0000 7fc595d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:40.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.447+0000 7fc595d9b640 1 -- 172.21.15.120:0/1578405434 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5880094d0 con 0x7fc59813f810 2024-02-06T21:41:40.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.449+0000 7fc595d9b640 1 --2- 172.21.15.120:0/1578405434 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc59813f810 0x7fc598136730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc58c00b410 tx=0x7fc58c00b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.449+0000 7fc586ffd640 10 client.?.objecter ms_handle_connect 0x7fc59813f810 2024-02-06T21:41:40.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.449+0000 7fc586ffd640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:40.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc586ffd640 1 -- 172.21.15.120:0/1578405434 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c011830 con 0x7fc59813f810 2024-02-06T21:41:40.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc586ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:40.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc586ffd640 10 monclient: dump: 2024-02-06T21:41:40.455 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.455 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.456 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.456 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.456 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.456 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.456 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.456 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.456 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.456 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc586ffd640 10 monclient: _finish_auth 0 2024-02-06T21:41:40.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc586ffd640 10 monclient: _check_auth_tickets 2024-02-06T21:41:40.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc586ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:40.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc586ffd640 1 -- 172.21.15.120:0/1578405434 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58c0119b0 con 0x7fc59813f810 2024-02-06T21:41:40.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc59cf82640 5 monclient: authenticate success, global_id 4354 2024-02-06T21:41:40.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc59cf82640 10 monclient: _renew_subs 2024-02-06T21:41:40.457 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc59cf82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:40.457 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc59cf82640 1 -- 172.21.15.120:0/1578405434 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc598144130 con 0x7fc59813f810 2024-02-06T21:41:40.457 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc586ffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:40.457 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc586ffd640 1 -- 172.21.15.120:0/1578405434 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c011d60 con 0x7fc59813f810 2024-02-06T21:41:40.457 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.457 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc586ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:40.457 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc586ffd640 10 monclient: dump: 2024-02-06T21:41:40.457 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.457 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.457 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.457 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.457 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.457 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.458 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.458 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.458 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.458 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.458 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc59cf82640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:40.458 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc59cf82640 10 monclient: _renew_subs 2024-02-06T21:41:40.458 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc59cf82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:40.458 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc59cf82640 1 -- 172.21.15.120:0/1578405434 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc598135a20 con 0x7fc59813f810 2024-02-06T21:41:40.458 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc594d99640 4 set_mon_vals no callback set 2024-02-06T21:41:40.458 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.453+0000 7fc59cf82640 1 librados: init done 2024-02-06T21:41:40.459 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.454+0000 7fc59cf82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.459 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.454+0000 7fc59cf82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.459 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.454+0000 7fc59cf82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:40.459 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.454+0000 7fc59cf82640 1 -- 172.21.15.120:0/1578405434 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59804ef90 con 0x7fc59813f810 2024-02-06T21:41:40.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.462+0000 7ffa01f23640 10 monclient: get_monmap_and_config 2024-02-06T21:41:40.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.462+0000 7ffa01f23640 10 monclient: build_initial_monmap 2024-02-06T21:41:40.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.462+0000 7ffa01f23640 10 monclient: monmap: 2024-02-06T21:41:40.464 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:40.464 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.464 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:40.463790+0000 2024-02-06T21:41:40.464 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:40.463790+0000 2024-02-06T21:41:40.464 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:40.464 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.464 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:40.464 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:40.464 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:40.464 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:40.464 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:40.464 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 1 Processor -- start 2024-02-06T21:41:40.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 1 -- start start 2024-02-06T21:41:40.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 10 monclient: init 2024-02-06T21:41:40.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:40.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-02-06T21:41:40.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9fc0a9c80 0x7ff9fc0aa060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 10 monclient(hunting): picked mon.noname-b con 0x7ff9fc0a9c80 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:40.467 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff9fc142bb0 0x7ff9fc142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.467 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 10 monclient(hunting): picked mon.noname-a con 0x7ff9fc142bb0 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:40.467 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9fc13f810 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:40.467 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9fc0aa5a0 con 0x7ff9fc13f810 2024-02-06T21:41:40.467 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.467 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9fc143510 con 0x7ff9fc142bb0 2024-02-06T21:41:40.467 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.467 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc142720 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.467 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:40.467 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.463+0000 7ffa01f23640 10 monclient(hunting): authenticate will time out at 1330.801446s 2024-02-06T21:41:40.467 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.464+0000 7ff9fb7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9fc0a9c80 0x7ff9fc0aa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.467 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.465+0000 7ff9fb7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9fc0a9c80 0x7ff9fc0aa060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57100/0 (socket says 172.21.15.120:57100) 2024-02-06T21:41:40.467 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.465+0000 7ff9fb7fe640 1 -- 172.21.15.120:0/2737439934 learned_addr learned my addr 172.21.15.120:0/2737439934 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:40.467 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.465+0000 7ff9fb7fe640 10 monclient(hunting): get_auth_request con 0x7ff9fc0a9c80 auth_method 0 2024-02-06T21:41:40.467 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.465+0000 7ff9fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.468 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.465+0000 7ff9fb7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.468 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.465+0000 7ff9fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.465+0000 7ff9fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.465+0000 7ff9fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.465+0000 7ff9fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.467+0000 7ff9fb7fe640 10 monclient(hunting): handle_auth_done global_id 4316 payload 274 2024-02-06T21:41:40.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.467+0000 7ff9fb7fe640 1 -- 172.21.15.120:0/2737439934 >> v2:172.21.15.120:3300/0 conn(0x7ff9fc142bb0 msgr2=0x7ff9fc142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:40.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.467+0000 7ff9fb7fe640 1 --2- 172.21.15.120:0/2737439934 >> v2:172.21.15.120:3300/0 conn(0x7ff9fc142bb0 0x7ff9fc142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.467+0000 7ff9fb7fe640 1 -- 172.21.15.120:0/2737439934 >> v1:172.21.15.120:6789/0 conn(0x7ff9fc13f810 legacy=0x7ff9fc13fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:40.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.467+0000 7ff9fb7fe640 10 monclient: _finish_hunting 0 2024-02-06T21:41:40.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.467+0000 7ff9fb7fe640 1 monclient: found mon.noname-b 2024-02-06T21:41:40.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.467+0000 7ff9fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:40.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.467+0000 7ff9fb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:40.469 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.467+0000 7ff9fb7fe640 1 -- 172.21.15.120:0/2737439934 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9fc0ab120 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.470 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.469+0000 7ff9fb7fe640 1 --2- 172.21.15.120:0/2737439934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9fc0a9c80 0x7ff9fc0aa060 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009e50 tx=0x7ff9ec02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af2c248d397978a1 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ff9fa7fc640 1 -- 172.21.15.120:0/2737439934 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec02ca70 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ff9fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ff9fa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:40.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ff9fa7fc640 10 monclient: dump: 2024-02-06T21:41:40.475 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.475 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.475 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.475 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.475 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.475 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.475 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.475 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.475 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.475 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ff9fa7fc640 10 monclient: _finish_auth 0 2024-02-06T21:41:40.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ff9fa7fc640 10 monclient: _check_auth_tickets 2024-02-06T21:41:40.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ff9fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:40.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ff9fa7fc640 1 -- 172.21.15.120:0/2737439934 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ec02cbd0 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ff9fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:40.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ff9fa7fc640 1 -- 172.21.15.120:0/2737439934 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec0343f0 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ff9fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ff9fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:40.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ff9fa7fc640 10 monclient: dump: 2024-02-06T21:41:40.476 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.476 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.476 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.476 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.477 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.477 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.477 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.477 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.477 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.477 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ffa01f23640 5 monclient: authenticate success, global_id 4316 2024-02-06T21:41:40.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ffa01f23640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:40.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.472+0000 7ffa01f23640 4 set_mon_vals no callback set 2024-02-06T21:41:40.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.473+0000 7ffa01f23640 10 monclient: shutdown 2024-02-06T21:41:40.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.473+0000 7ffa01f23640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:40.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.473+0000 7ffa01f23640 1 -- 172.21.15.120:0/2737439934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9fc0a9c80 msgr2=0x7ff9fc0aa060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.473+0000 7ffa01f23640 1 --2- 172.21.15.120:0/2737439934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9fc0a9c80 0x7ff9fc0aa060 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009e50 tx=0x7ff9ec02bed0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.473+0000 7ffa01f23640 1 -- 172.21.15.120:0/2737439934 shutdown_connections 2024-02-06T21:41:40.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.473+0000 7ffa01f23640 1 --2- 172.21.15.120:0/2737439934 >> v2:172.21.15.120:3300/0 conn(0x7ff9fc142bb0 0x7ff9fc142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.478 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.473+0000 7ffa01f23640 1 --2- 172.21.15.120:0/2737439934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9fc0a9c80 0x7ff9fc0aa060 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.478 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.473+0000 7ffa01f23640 1 -- 172.21.15.120:0/2737439934 >> 172.21.15.120:0/2737439934 conn(0x7ff9fc0b4ba0 msgr2=0x7ff9fc0b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:40.481 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.474+0000 7ffa01f23640 1 -- 172.21.15.120:0/2737439934 shutdown_connections 2024-02-06T21:41:40.481 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.474+0000 7ffa01f23640 1 -- 172.21.15.120:0/2737439934 wait complete. 2024-02-06T21:41:40.481 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.474+0000 7ffa01f23640 10 monclient: build_initial_monmap 2024-02-06T21:41:40.481 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.474+0000 7ffa01f23640 10 monclient: monmap: 2024-02-06T21:41:40.481 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:40.481 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:40.481 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:40.481 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:40.481 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:40.481 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.482 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:40.482 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:40.482 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:40.482 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.475+0000 7ffa01f23640 1 librados: starting msgr at 2024-02-06T21:41:40.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.475+0000 7ffa01f23640 1 librados: starting objecter 2024-02-06T21:41:40.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.475+0000 7ffa01f23640 1 Processor -- start 2024-02-06T21:41:40.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.475+0000 7ffa01f23640 1 -- start start 2024-02-06T21:41:40.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.475+0000 7ffa01f23640 1 librados: setting wanted keys 2024-02-06T21:41:40.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.475+0000 7ffa01f23640 1 librados: calling monclient init 2024-02-06T21:41:40.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.475+0000 7ffa01f23640 10 monclient: init 2024-02-06T21:41:40.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:40.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 10 monclient: _add_conns ranks=[2,0,1] 2024-02-06T21:41:40.482 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9fc0a9c80 0x7ff9fc13a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 10 monclient(hunting): picked mon.noname-c con 0x7ff9fc0a9c80 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:40.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9fc13f810 0x7ff9fc136730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 10 monclient(hunting): picked mon.noname-a con 0x7ff9fc13f810 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:40.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9fc142bb0 0x7ff9fc1354e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 10 monclient(hunting): picked mon.noname-b con 0x7ff9fc142bb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:40.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc1350e0 con 0x7ff9fc13f810 2024-02-06T21:41:40.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc136c70 con 0x7ff9fc142bb0 2024-02-06T21:41:40.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc143e50 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:40.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ffa01f23640 10 monclient(hunting): authenticate will time out at 1330.813907s 2024-02-06T21:41:40.483 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.476+0000 7ff9fb7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9fc0a9c80 0x7ff9fc13a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.484 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.477+0000 7ff9fb7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9fc0a9c80 0x7ff9fc13a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57112/0 (socket says 172.21.15.120:57112) 2024-02-06T21:41:40.484 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.477+0000 7ff9fb7fe640 1 -- 172.21.15.120:0/2116432464 learned_addr learned my addr 172.21.15.120:0/2116432464 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:40.484 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.477+0000 7ff9fb7fe640 10 monclient(hunting): get_auth_request con 0x7ff9fc0a9c80 auth_method 0 2024-02-06T21:41:40.484 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.477+0000 7ff9fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.484 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.477+0000 7ff9fb7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.484 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.477+0000 7ff9fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.484 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.477+0000 7ff9fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.484 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.477+0000 7ff9fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.484 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.477+0000 7ff9fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.484 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-02-06T21:41:40.485 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.480+0000 7ff9fb7fe640 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2024-02-06T21:41:40.485 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.480+0000 7ff9fb7fe640 1 -- 172.21.15.120:0/2116432464 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9fc142bb0 msgr2=0x7ff9fc1354e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.480+0000 7ff9fb7fe640 1 --2- 172.21.15.120:0/2116432464 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9fc142bb0 0x7ff9fc1354e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.480+0000 7ff9fb7fe640 1 -- 172.21.15.120:0/2116432464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9fc13f810 msgr2=0x7ff9fc136730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.480+0000 7ff9fb7fe640 1 --2- 172.21.15.120:0/2116432464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9fc13f810 0x7ff9fc136730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.480+0000 7ff9fb7fe640 10 monclient: _finish_hunting 0 2024-02-06T21:41:40.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.480+0000 7ff9fb7fe640 1 monclient: found mon.noname-c 2024-02-06T21:41:40.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.480+0000 7ff9fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:40.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.480+0000 7ff9fb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:41:40.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.480+0000 7ff9fb7fe640 1 -- 172.21.15.120:0/2116432464 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec009b30 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.482+0000 7ff9fb7fe640 1 --2- 172.21.15.120:0/2116432464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9fc0a9c80 0x7ff9fc13a3f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009e50 tx=0x7ff9ec004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.482+0000 7ff9e3fff640 10 client.?.objecter ms_handle_connect 0x7ff9fc0a9c80 2024-02-06T21:41:40.486 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.482+0000 7ff9e3fff640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:40.489 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ff9e3fff640 1 -- 172.21.15.120:0/2116432464 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec009510 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.489 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ff9e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.489 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ff9e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:40.489 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ff9e3fff640 10 monclient: dump: 2024-02-06T21:41:40.489 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.489 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.489 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.489 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.489 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.490 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.490 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.490 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.490 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.490 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.490 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ff9e3fff640 10 monclient: _finish_auth 0 2024-02-06T21:41:40.490 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ff9e3fff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:40.490 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ff9e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:40.490 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ff9e3fff640 1 -- 172.21.15.120:0/2116432464 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ec03e450 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.490 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ffa01f23640 5 monclient: authenticate success, global_id 4319 2024-02-06T21:41:40.490 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ff9e3fff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:40.490 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ff9e3fff640 1 -- 172.21.15.120:0/2116432464 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec03e7e0 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.490 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ff9e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.490 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ff9e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ff9e3fff640 10 monclient: dump: 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ffa01f23640 10 monclient: _renew_subs 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ffa01f23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ffa01f23640 1 -- 172.21.15.120:0/2116432464 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9fc144130 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ff9f9ffb640 4 set_mon_vals no callback set 2024-02-06T21:41:40.491 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ffa01f23640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:40.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ffa01f23640 10 monclient: _renew_subs 2024-02-06T21:41:40.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ffa01f23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:40.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ffa01f23640 1 -- 172.21.15.120:0/2116432464 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9fc144af0 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.486+0000 7ffa01f23640 1 librados: init done 2024-02-06T21:41:40.492 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.487+0000 7ffa01f23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.487+0000 7ffa01f23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.487+0000 7ffa01f23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:40.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.487+0000 7ffa01f23640 1 -- 172.21.15.120:0/2116432464 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9fc04ef90 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.504 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-02-06T21:41:40.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.503+0000 7f02aa7fc640 1 -- 172.21.15.120:0/3440068232 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f02a4076e90 con 0x7f02b413f830 2024-02-06T21:41:40.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.503+0000 7f02aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.503+0000 7f02aa7fc640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:40.576 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-02-06T21:41:40.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.606+0000 7f2bd65c4640 10 monclient: get_monmap_and_config 2024-02-06T21:41:40.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.606+0000 7f2bd65c4640 10 monclient: build_initial_monmap 2024-02-06T21:41:40.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.606+0000 7f2bd65c4640 10 monclient: monmap: 2024-02-06T21:41:40.608 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:40.608 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.608 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:40.607599+0000 2024-02-06T21:41:40.608 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:40.607599+0000 2024-02-06T21:41:40.608 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:40.608 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.608 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:40.609 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:40.609 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:40.609 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:40.609 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:40.609 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.612 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.607+0000 7f2bd65c4640 1 Processor -- start 2024-02-06T21:41:40.612 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.607+0000 7f2bd65c4640 1 -- start start 2024-02-06T21:41:40.612 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.607+0000 7f2bd65c4640 10 monclient: init 2024-02-06T21:41:40.612 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.608+0000 7f2bd65c4640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:40.612 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.608+0000 7f2bd65c4640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-02-06T21:41:40.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.608+0000 7f2bd65c4640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2bd00709d0 0x7f2bd0140930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.608+0000 7f2bd65c4640 10 monclient(hunting): picked mon.noname-c con 0x7f2bd00709d0 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:40.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.608+0000 7f2bd65c4640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2bd0070db0 0x7f2bd0140e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.608+0000 7f2bd65c4640 10 monclient(hunting): picked mon.noname-a con 0x7f2bd0070db0 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:40.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.608+0000 7f2bd65c4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2bd013d2a0 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:40.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.608+0000 7f2bd65c4640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd00726d0 con 0x7f2bd013d2a0 2024-02-06T21:41:40.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.608+0000 7f2bd65c4640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.608+0000 7f2bd65c4640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd01413b0 con 0x7f2bd0070db0 2024-02-06T21:41:40.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.608+0000 7f2bd65c4640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.608+0000 7f2bd65c4640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd0072940 con 0x7f2bd00709d0 2024-02-06T21:41:40.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.608+0000 7f2bd65c4640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:40.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.608+0000 7f2bd65c4640 10 monclient(hunting): authenticate will time out at 1330.946225s 2024-02-06T21:41:40.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.609+0000 7f2bcf7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2bd0070db0 0x7f2bd0140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.610+0000 7f2bcf7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2bd0070db0 0x7f2bd0140e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60362/0 (socket says 172.21.15.120:60362) 2024-02-06T21:41:40.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.610+0000 7f2bcf7fe640 1 -- 172.21.15.120:0/2641575861 learned_addr learned my addr 172.21.15.120:0/2641575861 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:40.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.610+0000 7f2bcf7fe640 10 monclient(hunting): get_auth_request con 0x7f2bd0070db0 auth_method 0 2024-02-06T21:41:40.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.610+0000 7f2bcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.610+0000 7f2bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.610+0000 7f2bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.610+0000 7f2bcf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.610+0000 7f2bcf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.610+0000 7f2bcf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.613+0000 7f2bcf7fe640 10 monclient(hunting): handle_auth_done global_id 4353 payload 274 2024-02-06T21:41:40.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.613+0000 7f2bcf7fe640 1 -- 172.21.15.120:0/2641575861 >> v2:172.21.15.135:3300/0 conn(0x7f2bd00709d0 msgr2=0x7f2bd0140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:40.616 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.613+0000 7f2bcf7fe640 1 --2- 172.21.15.120:0/2641575861 >> v2:172.21.15.135:3300/0 conn(0x7f2bd00709d0 0x7f2bd0140930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.616 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.613+0000 7f2bcf7fe640 1 -- 172.21.15.120:0/2641575861 >> v1:172.21.15.135:6789/0 conn(0x7f2bd013d2a0 legacy=0x7f2bd013d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:40.616 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.613+0000 7f2bcf7fe640 10 monclient: _finish_hunting 0 2024-02-06T21:41:40.616 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.613+0000 7f2bcf7fe640 1 monclient: found mon.noname-a 2024-02-06T21:41:40.616 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.613+0000 7f2bcf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:40.616 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.613+0000 7f2bcf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:40.616 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.613+0000 7f2bcf7fe640 1 -- 172.21.15.120:0/2641575861 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd0072c20 con 0x7f2bd0070db0 2024-02-06T21:41:40.617 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.616+0000 7f2bcf7fe640 1 --2- 172.21.15.120:0/2641575861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bd0070db0 0x7f2bd0140e70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2bd006d080 tx=0x7f2bc402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117287ada6895433 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.621 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.619+0000 7f2bceffd640 1 -- 172.21.15.120:0/2641575861 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4031800 con 0x7f2bd0070db0 2024-02-06T21:41:40.621 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.619+0000 7f2bceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.621 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.619+0000 7f2bceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:40.622 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.619+0000 7f2bceffd640 10 monclient: dump: 2024-02-06T21:41:40.622 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.622 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.622 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.622 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.622 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.622 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.622 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.622 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.622 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.622 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.622 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.619+0000 7f2bceffd640 10 monclient: _finish_auth 0 2024-02-06T21:41:40.622 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.619+0000 7f2bceffd640 10 monclient: _check_auth_tickets 2024-02-06T21:41:40.622 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.619+0000 7f2bceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:40.622 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.619+0000 7f2bd65c4640 5 monclient: authenticate success, global_id 4353 2024-02-06T21:41:40.623 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.619+0000 7f2bd65c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:40.624 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.622+0000 7f2bceffd640 1 -- 172.21.15.120:0/2641575861 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc4031960 con 0x7f2bd0070db0 2024-02-06T21:41:40.624 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.622+0000 7f2bceffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:40.624 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.622+0000 7f2bceffd640 1 -- 172.21.15.120:0/2641575861 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4031d30 con 0x7f2bd0070db0 2024-02-06T21:41:40.624 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.622+0000 7f2bceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.625 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.622+0000 7f2bceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:40.625 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.622+0000 7f2bceffd640 10 monclient: dump: 2024-02-06T21:41:40.625 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.625 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.625 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.625 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.625 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.625 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.625 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.625 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.625 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.625 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.625 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.622+0000 7f2bd65c4640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:40.625 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.622+0000 7f2bd65c4640 4 set_mon_vals no callback set 2024-02-06T21:41:40.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.623+0000 7f2bd65c4640 10 monclient: shutdown 2024-02-06T21:41:40.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.623+0000 7f2bd65c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:40.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.623+0000 7f2bd65c4640 1 -- 172.21.15.120:0/2641575861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bd0070db0 msgr2=0x7f2bd0140e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.623+0000 7f2bd65c4640 1 --2- 172.21.15.120:0/2641575861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bd0070db0 0x7f2bd0140e70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2bd006d080 tx=0x7f2bc402b8a0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.623+0000 7f2bd65c4640 1 -- 172.21.15.120:0/2641575861 shutdown_connections 2024-02-06T21:41:40.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.623+0000 7f2bd65c4640 1 --2- 172.21.15.120:0/2641575861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bd0070db0 0x7f2bd0140e70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.623+0000 7f2bd65c4640 1 --2- 172.21.15.120:0/2641575861 >> v2:172.21.15.135:3300/0 conn(0x7f2bd00709d0 0x7f2bd0140930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.623+0000 7f2bd65c4640 1 -- 172.21.15.120:0/2641575861 >> 172.21.15.120:0/2641575861 conn(0x7f2bd00b0bc0 msgr2=0x7f2bd00b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:40.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.624+0000 7f2bd65c4640 1 -- 172.21.15.120:0/2641575861 shutdown_connections 2024-02-06T21:41:40.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.624+0000 7f2bd65c4640 1 -- 172.21.15.120:0/2641575861 wait complete. 2024-02-06T21:41:40.628 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.626+0000 7f2bd65c4640 10 monclient: build_initial_monmap 2024-02-06T21:41:40.628 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.626+0000 7f2bd65c4640 10 monclient: monmap: 2024-02-06T21:41:40.628 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:40.628 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:40.628 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:40.628 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:40.628 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:40.628 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.628 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:40.628 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:40.628 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:40.628 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.631 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.626+0000 7f2bd65c4640 1 librados: starting msgr at 2024-02-06T21:41:40.631 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.626+0000 7f2bd65c4640 1 librados: starting objecter 2024-02-06T21:41:40.631 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.627+0000 7f2bd65c4640 1 Processor -- start 2024-02-06T21:41:40.631 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.627+0000 7f2bd65c4640 1 -- start start 2024-02-06T21:41:40.631 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.627+0000 7f2bd65c4640 1 librados: setting wanted keys 2024-02-06T21:41:40.631 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.627+0000 7f2bd65c4640 1 librados: calling monclient init 2024-02-06T21:41:40.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.627+0000 7f2bd65c4640 10 monclient: init 2024-02-06T21:41:40.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.628+0000 7f2bd65c4640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:40.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.628+0000 7f2bd65c4640 10 monclient: _add_conns ranks=[0,2,1] 2024-02-06T21:41:40.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.628+0000 7f2bd65c4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bd00709d0 0x7f2bd01443f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.628+0000 7f2bd65c4640 10 monclient(hunting): picked mon.noname-a con 0x7f2bd00709d0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:40.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.628+0000 7f2bd65c4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bd0070db0 0x7f2bd00a9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.629+0000 7f2bd65c4640 10 monclient(hunting): picked mon.noname-c con 0x7f2bd0070db0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:40.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.629+0000 7f2bd65c4640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2bd013d2a0 0x7f2bd01d9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.629+0000 7f2bd65c4640 10 monclient(hunting): picked mon.noname-b con 0x7f2bd013d2a0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:40.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.629+0000 7f2bd65c4640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.629+0000 7f2bd65c4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd01d8ca0 con 0x7f2bd00709d0 2024-02-06T21:41:40.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.629+0000 7f2bd65c4640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.629+0000 7f2bd65c4640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00aab90 con 0x7f2bd013d2a0 2024-02-06T21:41:40.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.629+0000 7f2bd65c4640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.629+0000 7f2bd65c4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00aae00 con 0x7f2bd0070db0 2024-02-06T21:41:40.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.629+0000 7f2bd65c4640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:40.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.629+0000 7f2bd65c4640 10 monclient(hunting): authenticate will time out at 1330.967200s 2024-02-06T21:41:40.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.629+0000 7f2bcffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bd00709d0 0x7f2bd01443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.634 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.630+0000 7f2bcffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bd00709d0 0x7f2bd01443f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60366/0 (socket says 172.21.15.120:60366) 2024-02-06T21:41:40.634 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.630+0000 7f2bcffff640 1 -- 172.21.15.120:0/471259868 learned_addr learned my addr 172.21.15.120:0/471259868 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:40.634 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.630+0000 7f2bcffff640 10 monclient(hunting): get_auth_request con 0x7f2bd00709d0 auth_method 0 2024-02-06T21:41:40.634 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.630+0000 7f2bcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.634 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.630+0000 7f2bcffff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.634 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.630+0000 7f2bcffff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.634 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.631+0000 7f2bcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.634 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.631+0000 7f2bcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.634 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.631+0000 7f2bcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.637 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.635+0000 7f2bcffff640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2024-02-06T21:41:40.637 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.635+0000 7f2bcffff640 1 -- 172.21.15.120:0/471259868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bd0070db0 msgr2=0x7f2bd00a9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.637 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.635+0000 7f2bcffff640 1 --2- 172.21.15.120:0/471259868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bd0070db0 0x7f2bd00a9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.637 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.635+0000 7f2bcffff640 1 -- 172.21.15.120:0/471259868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2bd013d2a0 msgr2=0x7f2bd01d9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.637 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.635+0000 7f2bcffff640 1 --2- 172.21.15.120:0/471259868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2bd013d2a0 0x7f2bd01d9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.637 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.635+0000 7f2bcffff640 10 monclient: _finish_hunting 0 2024-02-06T21:41:40.637 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.635+0000 7f2bcffff640 1 monclient: found mon.noname-a 2024-02-06T21:41:40.637 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.635+0000 7f2bcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:40.637 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.635+0000 7f2bcffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:40.638 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.635+0000 7f2bcffff640 1 -- 172.21.15.120:0/471259868 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc40094d0 con 0x7f2bd00709d0 2024-02-06T21:41:40.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.639+0000 7f2bcffff640 1 --2- 172.21.15.120:0/471259868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bd00709d0 0x7f2bd01443f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2bc000b4d0 tx=0x7f2bc000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.639+0000 7f2bccff9640 10 client.?.objecter ms_handle_connect 0x7f2bd00709d0 2024-02-06T21:41:40.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.639+0000 7f2bccff9640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:40.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bccff9640 1 -- 172.21.15.120:0/471259868 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0010870 con 0x7f2bd00709d0 2024-02-06T21:41:40.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bccff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:40.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bccff9640 10 monclient: dump: 2024-02-06T21:41:40.649 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.649 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.649 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.649 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.649 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.649 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.649 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.649 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.649 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.650 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bccff9640 10 monclient: _finish_auth 0 2024-02-06T21:41:40.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bccff9640 10 monclient: _check_auth_tickets 2024-02-06T21:41:40.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:40.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bccff9640 1 -- 172.21.15.120:0/471259868 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc00109d0 con 0x7f2bd00709d0 2024-02-06T21:41:40.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bd65c4640 5 monclient: authenticate success, global_id 4356 2024-02-06T21:41:40.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bd65c4640 10 monclient: _renew_subs 2024-02-06T21:41:40.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bd65c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:40.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bd65c4640 1 -- 172.21.15.120:0/471259868 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd00a9930 con 0x7f2bd00709d0 2024-02-06T21:41:40.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bccff9640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:40.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bccff9640 1 -- 172.21.15.120:0/471259868 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0010d80 con 0x7f2bd00709d0 2024-02-06T21:41:40.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:40.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bccff9640 10 monclient: dump: 2024-02-06T21:41:40.651 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.651 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.651 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.651 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.651 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.651 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.651 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.651 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.651 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.651 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.645+0000 7f2bce7fc640 4 set_mon_vals no callback set 2024-02-06T21:41:40.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.646+0000 7f2bd65c4640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:40.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.646+0000 7f2bd65c4640 10 monclient: _renew_subs 2024-02-06T21:41:40.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.646+0000 7f2bd65c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:40.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.646+0000 7f2bd65c4640 1 -- 172.21.15.120:0/471259868 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd01d9670 con 0x7f2bd00709d0 2024-02-06T21:41:40.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.646+0000 7f2bd65c4640 1 librados: init done 2024-02-06T21:41:40.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.648+0000 7f2bd65c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.648+0000 7f2bd65c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.648+0000 7f2bd65c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:40.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.648+0000 7f2bd65c4640 1 -- 172.21.15.120:0/471259868 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd004ef90 con 0x7f2bd00709d0 2024-02-06T21:41:40.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.668+0000 7f3a6a13b640 10 monclient: get_monmap_and_config 2024-02-06T21:41:40.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.668+0000 7f3a6a13b640 10 monclient: build_initial_monmap 2024-02-06T21:41:40.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.668+0000 7f3a6a13b640 10 monclient: monmap: 2024-02-06T21:41:40.670 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:40.670 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.670 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:40.669864+0000 2024-02-06T21:41:40.670 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:40.669864+0000 2024-02-06T21:41:40.670 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:40.670 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.670 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:40.670 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:40.671 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:40.671 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:40.671 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:40.671 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.672 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.669+0000 7f3a6a13b640 1 Processor -- start 2024-02-06T21:41:40.672 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.669+0000 7f3a6a13b640 1 -- start start 2024-02-06T21:41:40.672 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.669+0000 7f3a6a13b640 10 monclient: init 2024-02-06T21:41:40.672 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.670+0000 7f3a6a13b640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:40.672 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.670+0000 7f3a6a13b640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-02-06T21:41:40.672 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.670+0000 7f3a6a13b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a640b54b0 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:40.672 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.670+0000 7f3a6a13b640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3a6413ff40 0x7f3a640a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.672 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.670+0000 7f3a6a13b640 10 monclient(hunting): picked mon.noname-c con 0x7f3a6413ff40 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:40.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.670+0000 7f3a6a13b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a64142680 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:40.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.670+0000 7f3a6a13b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a64143200 con 0x7f3a640b54b0 2024-02-06T21:41:40.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.670+0000 7f3a6a13b640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a640ab720 con 0x7f3a64142680 2024-02-06T21:41:40.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.670+0000 7f3a6a13b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.670+0000 7f3a6a13b640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a640ab520 con 0x7f3a6413ff40 2024-02-06T21:41:40.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.670+0000 7f3a6a13b640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:40.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.670+0000 7f3a6a13b640 10 monclient(hunting): authenticate will time out at 1331.008478s 2024-02-06T21:41:40.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.679+0000 7feaff025640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-02-06T21:41:40.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.679+0000 7feaff025640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-02-06T21:41:40.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.679+0000 7feaff025640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:40.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.679+0000 7feaff025640 1 -- 172.21.15.120:0/907807584 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7feaf814df40 con 0x7feaf813f830 2024-02-06T21:41:40.684 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.682+0000 7f2bccff9640 1 -- 172.21.15.120:0/471259868 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2bc0023480 con 0x7f2bd00709d0 2024-02-06T21:41:40.685 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.682+0000 7f2bccff9640 1 --2- 172.21.15.120:0/471259868 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f2bbc06d220 0x7f2bbc06f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.685 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.683+0000 7f2bcf7fe640 1 --2- 172.21.15.120:0/471259868 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f2bbc06d220 0x7f2bbc06f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.685 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.683+0000 7f2bcf7fe640 10 monclient: get_auth_request con 0x7f2bbc06d220 auth_method 0 2024-02-06T21:41:40.685 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.683+0000 7f2bccff9640 10 client.4356.objecter ms_handle_connect 0x7f2bbc06d220 2024-02-06T21:41:40.685 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.683+0000 7f2bcf7fe640 1 --2- 172.21.15.120:0/471259868 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f2bbc06d220 0x7f2bbc06f6e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4036000 tx=0x7f2bc4030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.695 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.694+0000 7fcb6ed1c640 10 monclient: get_monmap_and_config 2024-02-06T21:41:40.695 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.694+0000 7fcb6ed1c640 10 monclient: build_initial_monmap 2024-02-06T21:41:40.695 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.694+0000 7fcb6ed1c640 10 monclient: monmap: 2024-02-06T21:41:40.695 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:40.695 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.695 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:40.695403+0000 2024-02-06T21:41:40.695 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:40.695403+0000 2024-02-06T21:41:40.696 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:40.696 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.696 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:40.696 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:40.696 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:40.696 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:40.696 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:40.696 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.694+0000 7fcb6ed1c640 1 Processor -- start 2024-02-06T21:41:40.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.694+0000 7fcb6ed1c640 1 -- start start 2024-02-06T21:41:40.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.694+0000 7fcb6ed1c640 10 monclient: init 2024-02-06T21:41:40.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.695+0000 7fcb6ed1c640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:40.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.695+0000 7fcb6ed1c640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-02-06T21:41:40.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.695+0000 7fcb6ed1c640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fcb680b54b0 0x7fcb680a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.695+0000 7fcb6ed1c640 10 monclient(hunting): picked mon.noname-c con 0x7fcb680b54b0 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:40.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.695+0000 7fcb6ed1c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb6813f830 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:40.702 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.695+0000 7fcb6ed1c640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcb68142680 0x7fcb68142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.702 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.695+0000 7fcb6ed1c640 10 monclient(hunting): picked mon.noname-a con 0x7fcb68142680 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:40.702 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.695+0000 7fcb6ed1c640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb680ab720 con 0x7fcb6813f830 2024-02-06T21:41:40.702 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.695+0000 7fcb6ed1c640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.702 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.695+0000 7fcb6ed1c640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb680ab520 con 0x7fcb68142680 2024-02-06T21:41:40.702 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.695+0000 7fcb6ed1c640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.702 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.695+0000 7fcb6ed1c640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb680b5890 con 0x7fcb680b54b0 2024-02-06T21:41:40.702 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.695+0000 7fcb6ed1c640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:40.702 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.695+0000 7fcb6ed1c640 10 monclient(hunting): authenticate will time out at 1331.033449s 2024-02-06T21:41:40.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.699+0000 7fc586ffd640 1 -- 172.21.15.120:0/1578405434 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc58c023430 con 0x7fc59813f810 2024-02-06T21:41:40.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.700+0000 7fc586ffd640 1 --2- 172.21.15.120:0/1578405434 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc57806d1f0 0x7fc57806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.700+0000 7fc59659c640 1 --2- 172.21.15.120:0/1578405434 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc57806d1f0 0x7fc57806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.700+0000 7fc59659c640 10 monclient: get_auth_request con 0x7fc57806d1f0 auth_method 0 2024-02-06T21:41:40.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.700+0000 7fc586ffd640 10 client.4354.objecter ms_handle_connect 0x7fc57806d1f0 2024-02-06T21:41:40.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.700+0000 7fc59659c640 1 --2- 172.21.15.120:0/1578405434 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc57806d1f0 0x7fc57806f6b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc58802b930 tx=0x7fc58803c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.704 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.703+0000 7fc586ffd640 1 -- 172.21.15.120:0/1578405434 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc58c0591e0 con 0x7fc59813f810 2024-02-06T21:41:40.704 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.703+0000 7fc586ffd640 10 client.4354.objecter ms_dispatch 0x7fc598000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:40.705 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.703+0000 7fc586ffd640 3 client.4354.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:40.705 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.703+0000 7fc586ffd640 3 client.4354.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:40.705 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.703+0000 7fc586ffd640 20 client.4354.objecter dump_active .. 0 homeless 2024-02-06T21:41:40.715 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.714+0000 7fc586ffd640 1 -- 172.21.15.120:0/1578405434 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc58c054d80 con 0x7fc59813f810 2024-02-06T21:41:40.715 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.714+0000 7fc586ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.715 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.714+0000 7fc586ffd640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:40.725 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.723+0000 7f3a62ffd640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3a6413ff40 0x7f3a640a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.726 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.724+0000 7f3a62ffd640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3a6413ff40 0x7f3a640a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38826/0 (socket says 172.21.15.120:38826) 2024-02-06T21:41:40.726 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.724+0000 7f3a62ffd640 1 -- 172.21.15.120:0/3733212249 learned_addr learned my addr 172.21.15.120:0/3733212249 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:40.726 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.724+0000 7f3a62ffd640 10 monclient(hunting): get_auth_request con 0x7f3a6413ff40 auth_method 0 2024-02-06T21:41:40.726 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.724+0000 7f3a62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.726 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.724+0000 7f3a62ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.726 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.724+0000 7f3a62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.726 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.725+0000 7fcb6cab8640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fcb680b54b0 0x7fcb680a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.727 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.726+0000 7f3a62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.727 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.726+0000 7f3a62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.727 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.726+0000 7f3a62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.728 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.726+0000 7fcb6cab8640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fcb680b54b0 0x7fcb680a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38834/0 (socket says 172.21.15.120:38834) 2024-02-06T21:41:40.728 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.726+0000 7fcb6cab8640 1 -- 172.21.15.120:0/2604579704 learned_addr learned my addr 172.21.15.120:0/2604579704 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:40.728 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.726+0000 7fcb6cab8640 10 monclient(hunting): get_auth_request con 0x7fcb680b54b0 auth_method 0 2024-02-06T21:41:40.728 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.726+0000 7fcb6cab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.728 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.726+0000 7fcb6cab8640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.728 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.726+0000 7fcb6cab8640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.729 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.727+0000 7fcb6cab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.729 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.727+0000 7fcb6cab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.729 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.727+0000 7fcb6cab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.730 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.729+0000 7fcb6cab8640 10 monclient(hunting): handle_auth_done global_id 4360 payload 274 2024-02-06T21:41:40.730 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.729+0000 7fcb6cab8640 1 -- 172.21.15.120:0/2604579704 >> v2:172.21.15.120:3300/0 conn(0x7fcb68142680 msgr2=0x7fcb68142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:40.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.729+0000 7fcb6cab8640 1 --2- 172.21.15.120:0/2604579704 >> v2:172.21.15.120:3300/0 conn(0x7fcb68142680 0x7fcb68142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.729+0000 7fcb6cab8640 1 -- 172.21.15.120:0/2604579704 >> v1:172.21.15.120:6789/0 conn(0x7fcb6813f830 legacy=0x7fcb6813fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:40.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.729+0000 7fcb6cab8640 10 monclient: _finish_hunting 0 2024-02-06T21:41:40.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.729+0000 7fcb6cab8640 1 monclient: found mon.noname-c 2024-02-06T21:41:40.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.729+0000 7fcb6cab8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:40.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.729+0000 7fcb6cab8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T21:41:40.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.729+0000 7fcb6cab8640 1 -- 172.21.15.120:0/2604579704 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb680b4800 con 0x7fcb680b54b0 2024-02-06T21:41:40.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.731+0000 7f3a62ffd640 10 monclient(hunting): handle_auth_done global_id 4357 payload 274 2024-02-06T21:41:40.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.731+0000 7f3a62ffd640 1 -- 172.21.15.120:0/3733212249 >> v1:172.21.15.135:6789/0 conn(0x7f3a64142680 legacy=0x7f3a64142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:40.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.731+0000 7f3a62ffd640 1 -- 172.21.15.120:0/3733212249 >> v1:172.21.15.120:6789/0 conn(0x7f3a640b54b0 legacy=0x7f3a6413f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:40.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.731+0000 7f3a62ffd640 10 monclient: _finish_hunting 0 2024-02-06T21:41:40.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.731+0000 7f3a62ffd640 1 monclient: found mon.noname-c 2024-02-06T21:41:40.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.731+0000 7f3a62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:40.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.731+0000 7f3a62ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T21:41:40.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.731+0000 7f3a62ffd640 1 -- 172.21.15.120:0/3733212249 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a640b4800 con 0x7f3a6413ff40 2024-02-06T21:41:40.734 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.733+0000 7fcb6cab8640 1 --2- 172.21.15.120:0/2604579704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb680b54b0 0x7fcb680a9860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcb6806c7b0 tx=0x7fcb5c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a7d9f51b0ef1a server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.735 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.734+0000 7f3a62ffd640 1 --2- 172.21.15.120:0/3733212249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a6413ff40 0x7f3a640a9860 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3a58009870 tx=0x7f3a5802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4a71cd27c1176f7 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb677fe640 1 -- 172.21.15.120:0/2604579704 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c031870 con 0x7fcb680b54b0 2024-02-06T21:41:40.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:40.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb677fe640 10 monclient: dump: 2024-02-06T21:41:40.744 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.744 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.744 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.744 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.744 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.744 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.744 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.744 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.744 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.744 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb677fe640 10 monclient: _finish_auth 0 2024-02-06T21:41:40.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb677fe640 10 monclient: _check_auth_tickets 2024-02-06T21:41:40.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:40.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb677fe640 1 -- 172.21.15.120:0/2604579704 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c0319d0 con 0x7fcb680b54b0 2024-02-06T21:41:40.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb6ed1c640 5 monclient: authenticate success, global_id 4360 2024-02-06T21:41:40.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb6ed1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:40.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb677fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:40.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb6ed1c640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:40.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb677fe640 1 -- 172.21.15.120:0/2604579704 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c031d60 con 0x7fcb680b54b0 2024-02-06T21:41:40.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb6ed1c640 4 set_mon_vals no callback set 2024-02-06T21:41:40.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:40.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb677fe640 10 monclient: dump: 2024-02-06T21:41:40.745 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.745 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.746 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.746 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.746 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.746 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.746 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.746 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.746 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.746 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb6ed1c640 10 monclient: shutdown 2024-02-06T21:41:40.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb6ed1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:40.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb6ed1c640 1 -- 172.21.15.120:0/2604579704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb680b54b0 msgr2=0x7fcb680a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.742+0000 7fcb6ed1c640 1 --2- 172.21.15.120:0/2604579704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb680b54b0 0x7fcb680a9860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcb6806c7b0 tx=0x7fcb5c02b910 comp rx=0 tx=0).stop 2024-02-06T21:41:40.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a627fc640 1 -- 172.21.15.120:0/3733212249 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58031870 con 0x7f3a6413ff40 2024-02-06T21:41:40.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:40.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a627fc640 10 monclient: dump: 2024-02-06T21:41:40.747 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.747 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.747 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.747 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.748 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.748 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.748 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.748 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.748 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.748 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.748 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a627fc640 10 monclient: _finish_auth 0 2024-02-06T21:41:40.748 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a627fc640 10 monclient: _check_auth_tickets 2024-02-06T21:41:40.748 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:40.748 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7fcb6ed1c640 1 -- 172.21.15.120:0/2604579704 shutdown_connections 2024-02-06T21:41:40.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7fcb6ed1c640 1 --2- 172.21.15.120:0/2604579704 >> v2:172.21.15.120:3300/0 conn(0x7fcb68142680 0x7fcb68142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7fcb6ed1c640 1 --2- 172.21.15.120:0/2604579704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb680b54b0 0x7fcb680a9860 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7fcb6ed1c640 1 -- 172.21.15.120:0/2604579704 >> 172.21.15.120:0/2604579704 conn(0x7fcb680b8660 msgr2=0x7fcb680b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:40.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a627fc640 1 -- 172.21.15.120:0/3733212249 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a580319d0 con 0x7f3a6413ff40 2024-02-06T21:41:40.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a627fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:40.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a627fc640 1 -- 172.21.15.120:0/3733212249 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58031d60 con 0x7f3a6413ff40 2024-02-06T21:41:40.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:40.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a627fc640 10 monclient: dump: 2024-02-06T21:41:40.750 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.750 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.750 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.750 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.750 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.750 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.750 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.750 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.750 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.750 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.750 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a6a13b640 5 monclient: authenticate success, global_id 4357 2024-02-06T21:41:40.750 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a6a13b640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:40.750 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7f3a6a13b640 4 set_mon_vals no callback set 2024-02-06T21:41:40.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7fcb6ed1c640 1 -- 172.21.15.120:0/2604579704 shutdown_connections 2024-02-06T21:41:40.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7fcb6ed1c640 1 -- 172.21.15.120:0/2604579704 wait complete. 2024-02-06T21:41:40.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7fcb6ed1c640 10 monclient: build_initial_monmap 2024-02-06T21:41:40.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.743+0000 7fcb6ed1c640 10 monclient: monmap: 2024-02-06T21:41:40.751 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:40.751 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:40.751 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:40.751 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:40.751 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:40.751 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.751 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:40.751 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:40.751 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:40.752 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7f3a6a13b640 10 monclient: shutdown 2024-02-06T21:41:40.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7f3a6a13b640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:40.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7f3a6a13b640 1 -- 172.21.15.120:0/3733212249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a6413ff40 msgr2=0x7f3a640a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7f3a6a13b640 1 --2- 172.21.15.120:0/3733212249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a6413ff40 0x7f3a640a9860 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3a58009870 tx=0x7f3a5802b910 comp rx=0 tx=0).stop 2024-02-06T21:41:40.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7f3a6a13b640 1 -- 172.21.15.120:0/3733212249 shutdown_connections 2024-02-06T21:41:40.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7f3a6a13b640 1 --2- 172.21.15.120:0/3733212249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a6413ff40 0x7f3a640a9860 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7f3a6a13b640 1 -- 172.21.15.120:0/3733212249 >> 172.21.15.120:0/3733212249 conn(0x7f3a640b8660 msgr2=0x7f3a640b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:40.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7fcb6ed1c640 1 librados: starting msgr at 2024-02-06T21:41:40.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7fcb6ed1c640 1 librados: starting objecter 2024-02-06T21:41:40.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7f3a6a13b640 1 -- 172.21.15.120:0/3733212249 shutdown_connections 2024-02-06T21:41:40.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7f3a6a13b640 1 -- 172.21.15.120:0/3733212249 wait complete. 2024-02-06T21:41:40.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7fcb6ed1c640 1 Processor -- start 2024-02-06T21:41:40.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7fcb6ed1c640 1 -- start start 2024-02-06T21:41:40.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7fcb6ed1c640 1 librados: setting wanted keys 2024-02-06T21:41:40.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7fcb6ed1c640 1 librados: calling monclient init 2024-02-06T21:41:40.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.744+0000 7fcb6ed1c640 10 monclient: init 2024-02-06T21:41:40.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:40.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T21:41:40.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb680b54b0 0x7fcb6814c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7f3a6a13b640 10 monclient: build_initial_monmap 2024-02-06T21:41:40.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7f3a6a13b640 10 monclient: monmap: 2024-02-06T21:41:40.755 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:40.755 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:40.755 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:40.755 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:40.755 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:40.755 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.756 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:40.756 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:40.756 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:40.756 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 10 monclient(hunting): picked mon.noname-b con 0x7fcb680b54b0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:40.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb6813f830 0x7fcb68134620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 10 monclient(hunting): picked mon.noname-c con 0x7fcb6813f830 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:40.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb68142680 0x7fcb68137fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 10 monclient(hunting): picked mon.noname-a con 0x7fcb68142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:40.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb68138520 con 0x7fcb68142680 2024-02-06T21:41:40.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb68138790 con 0x7fcb680b54b0 2024-02-06T21:41:40.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb68133b60 con 0x7fcb6813f830 2024-02-06T21:41:40.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:40.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6ed1c640 10 monclient(hunting): authenticate will time out at 1331.083113s 2024-02-06T21:41:40.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.745+0000 7fcb6cab8640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb680b54b0 0x7fcb6814c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7f3a6a13b640 1 librados: starting msgr at 2024-02-06T21:41:40.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7f3a6a13b640 1 librados: starting objecter 2024-02-06T21:41:40.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7fcb6cab8640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb680b54b0 0x7fcb6814c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38840/0 (socket says 172.21.15.120:38840) 2024-02-06T21:41:40.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7fcb6cab8640 1 -- 172.21.15.120:0/192166374 learned_addr learned my addr 172.21.15.120:0/192166374 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:40.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7fcb6cab8640 10 monclient(hunting): get_auth_request con 0x7fcb680b54b0 auth_method 0 2024-02-06T21:41:40.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7fcb6cab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7fcb6cab8640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7fcb6cab8640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7f3a6a13b640 1 Processor -- start 2024-02-06T21:41:40.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7f3a6a13b640 1 -- start start 2024-02-06T21:41:40.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7f3a6a13b640 1 librados: setting wanted keys 2024-02-06T21:41:40.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7f3a6a13b640 1 librados: calling monclient init 2024-02-06T21:41:40.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7f3a6a13b640 10 monclient: init 2024-02-06T21:41:40.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7fcb6cab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7fcb6cab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.746+0000 7fcb6cab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.747+0000 7f3a6a13b640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:40.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.747+0000 7f3a6a13b640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T21:41:40.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.747+0000 7f3a6a13b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a640b54b0 0x7f3a640b7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.747+0000 7f3a6a13b640 10 monclient(hunting): picked mon.noname-c con 0x7f3a640b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:40.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.747+0000 7f3a6a13b640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a6413ff40 0x7f3a64137a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.747+0000 7f3a6a13b640 10 monclient(hunting): picked mon.noname-b con 0x7f3a6413ff40 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:40.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.747+0000 7f3a6a13b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a64142680 0x7f3a64135570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.748+0000 7f3a6a13b640 10 monclient(hunting): picked mon.noname-a con 0x7f3a64142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:40.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.748+0000 7f3a6a13b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.748+0000 7f3a6a13b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a64136e40 con 0x7f3a64142680 2024-02-06T21:41:40.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.748+0000 7f3a6a13b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.748+0000 7f3a6a13b640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a64137020 con 0x7f3a6413ff40 2024-02-06T21:41:40.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.748+0000 7f3a6a13b640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.748+0000 7f3a6a13b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a64137300 con 0x7f3a640b54b0 2024-02-06T21:41:40.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.748+0000 7f3a6a13b640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:40.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.748+0000 7f3a6a13b640 10 monclient(hunting): authenticate will time out at 1331.085664s 2024-02-06T21:41:40.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.749+0000 7fcb6cab8640 10 monclient(hunting): handle_auth_done global_id 4363 payload 824 2024-02-06T21:41:40.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.749+0000 7fcb6cab8640 1 -- 172.21.15.120:0/192166374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb6813f830 msgr2=0x7fcb68134620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.749+0000 7fcb6cab8640 1 --2- 172.21.15.120:0/192166374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb6813f830 0x7fcb68134620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.749+0000 7fcb6cab8640 1 -- 172.21.15.120:0/192166374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb68142680 msgr2=0x7fcb68137fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.749+0000 7fcb6cab8640 1 --2- 172.21.15.120:0/192166374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb68142680 0x7fcb68137fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.749+0000 7fcb6cab8640 10 monclient: _finish_hunting 0 2024-02-06T21:41:40.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.749+0000 7fcb6cab8640 1 monclient: found mon.noname-b 2024-02-06T21:41:40.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.749+0000 7fcb6cab8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:40.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.749+0000 7fcb6cab8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:40.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.749+0000 7fcb6cab8640 1 -- 172.21.15.120:0/192166374 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c0094d0 con 0x7fcb680b54b0 2024-02-06T21:41:40.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.751+0000 7fcb6cab8640 1 --2- 172.21.15.120:0/192166374 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb680b54b0 0x7fcb6814c440 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c000c00 tx=0x7fcb5c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.751+0000 7fcb657fa640 10 client.?.objecter ms_handle_connect 0x7fcb680b54b0 2024-02-06T21:41:40.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.751+0000 7fcb657fa640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:40.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.751+0000 7f3a62ffd640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a6413ff40 0x7f3a64137a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.752+0000 7f3a62ffd640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a6413ff40 0x7f3a64137a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38842/0 (socket says 172.21.15.120:38842) 2024-02-06T21:41:40.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.752+0000 7f3a62ffd640 1 -- 172.21.15.120:0/2492854313 learned_addr learned my addr 172.21.15.120:0/2492854313 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:40.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.752+0000 7f3a62ffd640 10 monclient(hunting): get_auth_request con 0x7f3a6413ff40 auth_method 0 2024-02-06T21:41:40.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.752+0000 7f3a62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:40.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.752+0000 7f3a62ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:40.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.752+0000 7f3a62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:40.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.753+0000 7f3a62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:40.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.753+0000 7f3a62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:40.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.753+0000 7f3a62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:40.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.757+0000 7f3a62ffd640 10 monclient(hunting): handle_auth_done global_id 4366 payload 824 2024-02-06T21:41:40.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.757+0000 7f3a62ffd640 1 -- 172.21.15.120:0/2492854313 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a640b54b0 msgr2=0x7f3a640b7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.757+0000 7f3a62ffd640 1 --2- 172.21.15.120:0/2492854313 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a640b54b0 0x7f3a640b7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.757+0000 7f3a62ffd640 1 -- 172.21.15.120:0/2492854313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a64142680 msgr2=0x7f3a64135570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.757+0000 7f3a62ffd640 1 --2- 172.21.15.120:0/2492854313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a64142680 0x7f3a64135570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.757+0000 7f3a62ffd640 10 monclient: _finish_hunting 0 2024-02-06T21:41:40.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.757+0000 7f3a62ffd640 1 monclient: found mon.noname-b 2024-02-06T21:41:40.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.757+0000 7f3a62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:40.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.757+0000 7f3a62ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:40.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.757+0000 7f3a62ffd640 1 -- 172.21.15.120:0/2492854313 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a580094d0 con 0x7f3a6413ff40 2024-02-06T21:41:40.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.761+0000 7fcb657fa640 1 -- 172.21.15.120:0/192166374 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c031840 con 0x7fcb680b54b0 2024-02-06T21:41:40.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.761+0000 7fcb657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.761+0000 7fcb657fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:40.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.761+0000 7fcb657fa640 10 monclient: dump: 2024-02-06T21:41:40.767 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.767 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.768 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.768 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.768 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.768 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.768 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.768 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.768 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.768 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.761+0000 7fcb657fa640 10 monclient: _finish_auth 0 2024-02-06T21:41:40.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.761+0000 7fcb657fa640 10 monclient: _check_auth_tickets 2024-02-06T21:41:40.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.761+0000 7fcb657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:40.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.761+0000 7fcb657fa640 1 -- 172.21.15.120:0/192166374 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c0443f0 con 0x7fcb680b54b0 2024-02-06T21:41:40.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.761+0000 7fcb657fa640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:40.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.761+0000 7fcb66ffd640 4 set_mon_vals no callback set 2024-02-06T21:41:40.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.761+0000 7f3a62ffd640 1 --2- 172.21.15.120:0/2492854313 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a6413ff40 0x7f3a64137a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3a58000c00 tx=0x7f3a580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.761+0000 7f3a4bfff640 10 client.?.objecter ms_handle_connect 0x7f3a6413ff40 2024-02-06T21:41:40.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.761+0000 7f3a4bfff640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:40.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.762+0000 7fcb6ed1c640 5 monclient: authenticate success, global_id 4363 2024-02-06T21:41:40.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.762+0000 7fcb657fa640 1 -- 172.21.15.120:0/192166374 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0447a0 con 0x7fcb680b54b0 2024-02-06T21:41:40.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.762+0000 7fcb6ed1c640 10 monclient: _renew_subs 2024-02-06T21:41:40.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.762+0000 7fcb6ed1c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:40.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.762+0000 7fcb6ed1c640 1 -- 172.21.15.120:0/192166374 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb68133e40 con 0x7fcb680b54b0 2024-02-06T21:41:40.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.762+0000 7fcb6ed1c640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:40.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.762+0000 7fcb6ed1c640 10 monclient: _renew_subs 2024-02-06T21:41:40.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.762+0000 7fcb6ed1c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:40.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.762+0000 7fcb6ed1c640 1 -- 172.21.15.120:0/192166374 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb68138e20 con 0x7fcb680b54b0 2024-02-06T21:41:40.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.762+0000 7fcb6ed1c640 1 librados: init done 2024-02-06T21:41:40.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.762+0000 7fcb657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.762+0000 7fcb657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:40.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.762+0000 7fcb657fa640 10 monclient: dump: 2024-02-06T21:41:40.771 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.771 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.771 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.771 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.771 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.771 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.771 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.771 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.771 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.771 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.763+0000 7fcb6ed1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.763+0000 7fcb6ed1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.763+0000 7fcb6ed1c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:40.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.763+0000 7fcb6ed1c640 1 -- 172.21.15.120:0/192166374 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb68134360 con 0x7fcb680b54b0 2024-02-06T21:41:40.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a4bfff640 1 -- 172.21.15.120:0/2492854313 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5803ad30 con 0x7f3a6413ff40 2024-02-06T21:41:40.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a4bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:40.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a4bfff640 10 monclient: dump: 2024-02-06T21:41:40.775 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.775 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.775 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.775 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.775 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.775 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.775 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.775 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.775 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.775 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a4bfff640 10 monclient: _finish_auth 0 2024-02-06T21:41:40.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a4bfff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:40.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:40.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a4bfff640 1 -- 172.21.15.120:0/2492854313 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5803ae90 con 0x7f3a6413ff40 2024-02-06T21:41:40.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a4bfff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:40.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a4bfff640 1 -- 172.21.15.120:0/2492854313 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58044630 con 0x7f3a6413ff40 2024-02-06T21:41:40.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:40.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a4bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:40.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a4bfff640 10 monclient: dump: 2024-02-06T21:41:40.776 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:40.776 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.776 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.776 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:40.776 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:40.776 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.776 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:40.776 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:40.777 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:40.777 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a61ffb640 4 set_mon_vals no callback set 2024-02-06T21:41:40.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a6a13b640 5 monclient: authenticate success, global_id 4366 2024-02-06T21:41:40.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a6a13b640 10 monclient: _renew_subs 2024-02-06T21:41:40.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a6a13b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:40.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.766+0000 7f3a6a13b640 1 -- 172.21.15.120:0/2492854313 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a641375e0 con 0x7f3a6413ff40 2024-02-06T21:41:40.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.767+0000 7f3a6a13b640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:40.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.767+0000 7f3a6a13b640 10 monclient: _renew_subs 2024-02-06T21:41:40.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.767+0000 7f3a6a13b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:40.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.767+0000 7f3a6a13b640 1 -- 172.21.15.120:0/2492854313 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a64137f40 con 0x7f3a6413ff40 2024-02-06T21:41:40.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.767+0000 7f3a6a13b640 1 librados: init done 2024-02-06T21:41:40.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.768+0000 7f3a6a13b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.768+0000 7f3a6a13b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.768+0000 7f3a6a13b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:40.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.768+0000 7f3a6a13b640 1 -- 172.21.15.120:0/2492854313 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6404ef90 con 0x7f3a6413ff40 2024-02-06T21:41:40.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.770+0000 7f02bbe86640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-02-06T21:41:40.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.770+0000 7f02bbe86640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-02-06T21:41:40.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.770+0000 7f02bbe86640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:40.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.770+0000 7f02bbe86640 1 -- 172.21.15.120:0/3440068232 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f02b404ef90 con 0x7f02b413f830 2024-02-06T21:41:40.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.783+0000 7ff9e3fff640 1 -- 172.21.15.120:0/2116432464 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff9ec03e940 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.785 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.783+0000 7ff9e3fff640 1 --2- 172.21.15.120:0/2116432464 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff9e806d1f0 0x7ff9e806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.785 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.784+0000 7ff9faffd640 1 --2- 172.21.15.120:0/2116432464 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff9e806d1f0 0x7ff9e806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:40.785 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.784+0000 7ff9faffd640 10 monclient: get_auth_request con 0x7ff9e806d1f0 auth_method 0 2024-02-06T21:41:40.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.784+0000 7ff9faffd640 1 --2- 172.21.15.120:0/2116432464 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff9e806d1f0 0x7ff9e806f6b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc13abf0 tx=0x7ff9f0006bf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:40.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.784+0000 7ff9e3fff640 10 client.4319.objecter ms_handle_connect 0x7ff9e806d1f0 2024-02-06T21:41:40.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.795+0000 7ff9e3fff640 1 -- 172.21.15.120:0/2116432464 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff9ec07ca10 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.796+0000 7ff9e3fff640 10 client.4319.objecter ms_dispatch 0x7ff9fc000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:40.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.796+0000 7ff9e3fff640 3 client.4319.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:40.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.796+0000 7ff9e3fff640 3 client.4319.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:40.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.796+0000 7ff9e3fff640 20 client.4319.objecter dump_active .. 0 homeless 2024-02-06T21:41:40.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.796+0000 7ff9e3fff640 1 -- 172.21.15.120:0/2116432464 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff9ec0785b0 con 0x7ff9fc0a9c80 2024-02-06T21:41:40.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.796+0000 7ff9e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:40.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.796+0000 7ff9e3fff640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:40.819 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-02-06T21:41:40.820 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.817+0000 7feaf57fa640 1 -- 172.21.15.120:0/907807584 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feaf814df40 con 0x7feaf813f830 2024-02-06T21:41:40.820 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.817+0000 7feaf57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-02-06T21:41:40.820 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.817+0000 7feaf57fa640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:40.821 INFO:teuthology.orchestra.run.smithi120.stdout:73014444039 2024-02-06T21:41:40.822 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.819+0000 7feaff025640 10 librados: watch_flush enter 2024-02-06T21:41:40.822 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.819+0000 7feaff025640 10 librados: watch_flush exit 2024-02-06T21:41:40.822 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.819+0000 7feaff025640 20 client.4307.objecter shutdown clearing up homeless session... 2024-02-06T21:41:40.822 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.819+0000 7feaff025640 10 client.4307.objecter successfully canceled tick 2024-02-06T21:41:40.822 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.819+0000 7feaff025640 1 -- 172.21.15.120:0/907807584 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7feadc06d1f0 msgr2=0x7feadc06f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.822 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.819+0000 7feaff025640 1 --2- 172.21.15.120:0/907807584 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7feadc06d1f0 0x7feadc06f6b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feaec036000 tx=0x7feaec03c040 comp rx=0 tx=0).stop 2024-02-06T21:41:40.823 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.819+0000 7feaff025640 10 monclient: shutdown 2024-02-06T21:41:40.823 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.819+0000 7feaff025640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:40.823 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.819+0000 7feaff025640 1 -- 172.21.15.120:0/907807584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf813f830 msgr2=0x7feaf81324d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:40.823 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.819+0000 7feaff025640 1 --2- 172.21.15.120:0/907807584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf813f830 0x7feaf81324d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7feae400c400 tx=0x7feae400c8c0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.823 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.820+0000 7feaff025640 1 -- 172.21.15.120:0/907807584 shutdown_connections 2024-02-06T21:41:40.823 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.820+0000 7feaff025640 1 --2- 172.21.15.120:0/907807584 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7feadc06d1f0 0x7feadc06f6b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.823 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.820+0000 7feaff025640 1 --2- 172.21.15.120:0/907807584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feaf8142680 0x7feaf8132a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.823 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.820+0000 7feaff025640 1 --2- 172.21.15.120:0/907807584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feaf813f830 0x7feaf81324d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.823 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.820+0000 7feaff025640 1 --2- 172.21.15.120:0/907807584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feaf80b54b0 0x7feaf8135750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:40.823 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.820+0000 7feaff025640 1 -- 172.21.15.120:0/907807584 >> 172.21.15.120:0/907807584 conn(0x7feaf80b8660 msgr2=0x7feaf80b32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:40.823 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.820+0000 7feaff025640 1 -- 172.21.15.120:0/907807584 shutdown_connections 2024-02-06T21:41:40.823 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.820+0000 7feaff025640 1 -- 172.21.15.120:0/907807584 wait complete. 2024-02-06T21:41:40.823 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.820+0000 7feaff025640 1 librados: shutdown 2024-02-06T21:41:40.834 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444039 for osd.4 2024-02-06T21:41:40.914 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.912+0000 7fc59cf82640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-02-06T21:41:40.914 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.912+0000 7fc59cf82640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-02-06T21:41:40.914 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.912+0000 7fc59cf82640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:40.914 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.912+0000 7fc59cf82640 1 -- 172.21.15.120:0/1578405434 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc5980b65c0 con 0x7fc59813f810 2024-02-06T21:41:40.943 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.941+0000 7f61b8d04640 10 monclient: get_monmap_and_config 2024-02-06T21:41:40.943 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.941+0000 7f61b8d04640 10 monclient: build_initial_monmap 2024-02-06T21:41:40.943 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.941+0000 7f61b8d04640 10 monclient: monmap: 2024-02-06T21:41:40.943 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:40.943 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:40.943 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:40.943103+0000 2024-02-06T21:41:40.943 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:40.943103+0000 2024-02-06T21:41:40.944 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:40.944 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:40.944 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:40.944 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:40.944 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:40.944 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:40.944 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:40.944 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:40.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.942+0000 7f61b8d04640 1 Processor -- start 2024-02-06T21:41:40.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.942+0000 7f61b8d04640 1 -- start start 2024-02-06T21:41:40.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.942+0000 7f61b8d04640 10 monclient: init 2024-02-06T21:41:40.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.943+0000 7f61b8d04640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:40.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.943+0000 7f61b8d04640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-02-06T21:41:40.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.943+0000 7f61b8d04640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f61b40b54b0 0x7f61b40a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:40.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.943+0000 7f61b8d04640 10 monclient(hunting): picked mon.noname-c con 0x7f61b40b54b0 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:40.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.943+0000 7f61b8d04640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61b413f830 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:40.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.943+0000 7f61b8d04640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f61b4142680 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:40.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.943+0000 7f61b8d04640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b4143200 con 0x7f61b4142680 2024-02-06T21:41:40.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.943+0000 7f61b8d04640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b40ab720 con 0x7f61b413f830 2024-02-06T21:41:40.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.943+0000 7f61b8d04640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:40.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.943+0000 7f61b8d04640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b40ab520 con 0x7f61b40b54b0 2024-02-06T21:41:40.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.943+0000 7f61b8d04640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:40.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:40.943+0000 7f61b8d04640 10 monclient(hunting): authenticate will time out at 1331.280775s 2024-02-06T21:41:41.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.012+0000 7fcb657fa640 1 -- 172.21.15.120:0/192166374 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcb5c04d450 con 0x7fcb680b54b0 2024-02-06T21:41:41.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.012+0000 7fcb657fa640 1 --2- 172.21.15.120:0/192166374 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fcb4c06d220 0x7fcb4c06f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:41.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.012+0000 7fcb67fff640 1 --2- 172.21.15.120:0/192166374 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fcb4c06d220 0x7fcb4c06f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:41.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.012+0000 7fcb67fff640 10 monclient: get_auth_request con 0x7fcb4c06d220 auth_method 0 2024-02-06T21:41:41.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.013+0000 7fcb657fa640 10 client.4363.objecter ms_handle_connect 0x7fcb4c06d220 2024-02-06T21:41:41.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.013+0000 7fcb67fff640 1 --2- 172.21.15.120:0/192166374 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fcb4c06d220 0x7fcb4c06f6e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcb6813d390 tx=0x7fcb54007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:41.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.016+0000 7fcb657fa640 1 -- 172.21.15.120:0/192166374 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcb5c02d040 con 0x7fcb680b54b0 2024-02-06T21:41:41.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.016+0000 7fcb657fa640 10 client.4363.objecter ms_dispatch 0x7fcb68000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:41.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.016+0000 7fcb657fa640 3 client.4363.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:41.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.016+0000 7fcb657fa640 3 client.4363.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:41.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.016+0000 7fcb657fa640 20 client.4363.objecter dump_active .. 0 homeless 2024-02-06T21:41:41.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.026+0000 7fcb657fa640 1 -- 172.21.15.120:0/192166374 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcb5c0d88c0 con 0x7fcb680b54b0 2024-02-06T21:41:41.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.026+0000 7fcb657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:41.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.026+0000 7fcb657fa640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:41.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.030+0000 7fc586ffd640 1 -- 172.21.15.120:0/1578405434 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc58c015070 con 0x7fc59813f810 2024-02-06T21:41:41.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.030+0000 7fc586ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-02-06T21:41:41.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.030+0000 7fc586ffd640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:41.031 INFO:teuthology.orchestra.run.smithi120.stdout:77309411335 2024-02-06T21:41:41.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 10 librados: watch_flush enter 2024-02-06T21:41:41.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 10 librados: watch_flush exit 2024-02-06T21:41:41.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 20 client.4354.objecter shutdown clearing up homeless session... 2024-02-06T21:41:41.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 10 client.4354.objecter successfully canceled tick 2024-02-06T21:41:41.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 1 -- 172.21.15.120:0/1578405434 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc57806d1f0 msgr2=0x7fc57806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 1 --2- 172.21.15.120:0/1578405434 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc57806d1f0 0x7fc57806f6b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc58802b930 tx=0x7fc58803c040 comp rx=0 tx=0).stop 2024-02-06T21:41:41.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 10 monclient: shutdown 2024-02-06T21:41:41.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:41.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 1 -- 172.21.15.120:0/1578405434 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc59813f810 msgr2=0x7fc598136730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 1 --2- 172.21.15.120:0/1578405434 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc59813f810 0x7fc598136730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc58c00b410 tx=0x7fc58c00b8d0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 1 -- 172.21.15.120:0/1578405434 shutdown_connections 2024-02-06T21:41:41.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 1 --2- 172.21.15.120:0/1578405434 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc57806d1f0 0x7fc57806f6b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 1 --2- 172.21.15.120:0/1578405434 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc59813f810 0x7fc598136730 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 1 --2- 172.21.15.120:0/1578405434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc598142bb0 0x7fc5981354e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 1 --2- 172.21.15.120:0/1578405434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5980a9c80 0x7fc59813a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 1 -- 172.21.15.120:0/1578405434 >> 172.21.15.120:0/1578405434 conn(0x7fc5980b4ba0 msgr2=0x7fc59806d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:41.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 1 -- 172.21.15.120:0/1578405434 shutdown_connections 2024-02-06T21:41:41.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 1 -- 172.21.15.120:0/1578405434 wait complete. 2024-02-06T21:41:41.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.032+0000 7fc59cf82640 1 librados: shutdown 2024-02-06T21:41:41.044 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411336 got 77309411335 for osd.1 2024-02-06T21:41:41.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.058+0000 7f2bccff9640 1 -- 172.21.15.120:0/471259868 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2bc0059480 con 0x7f2bd00709d0 2024-02-06T21:41:41.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.058+0000 7f2bccff9640 10 client.4356.objecter ms_dispatch 0x7f2bd0000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:41.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.058+0000 7f2bccff9640 3 client.4356.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:41.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.058+0000 7f2bccff9640 3 client.4356.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:41.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.059+0000 7f2bccff9640 20 client.4356.objecter dump_active .. 0 homeless 2024-02-06T21:41:41.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.059+0000 7f2bccff9640 1 -- 172.21.15.120:0/471259868 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2bc0055020 con 0x7f2bd00709d0 2024-02-06T21:41:41.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.059+0000 7f2bccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:41.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.059+0000 7f2bccff9640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:41.063 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.062+0000 7f61b2d9d640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f61b4142680 0x7f61b4142af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:53122/0 (socket says 172.21.15.120:53122) 2024-02-06T21:41:41.064 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.062+0000 7f61b2d9d640 1 -- 172.21.15.120:0/448305521 learned_addr learned my addr 172.21.15.120:0/448305521 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:41:41.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.072+0000 7ffa01f23640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-02-06T21:41:41.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.072+0000 7ffa01f23640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-02-06T21:41:41.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.072+0000 7ffa01f23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:41.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.072+0000 7ffa01f23640 1 -- 172.21.15.120:0/2116432464 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff9fc135350 con 0x7ff9fc0a9c80 2024-02-06T21:41:41.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.088+0000 7f02aa7fc640 1 -- 172.21.15.120:0/3440068232 <== mon.0 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f02a407a550 con 0x7f02b413f830 2024-02-06T21:41:41.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.088+0000 7f02aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-02-06T21:41:41.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.088+0000 7f02aa7fc640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:41.090 INFO:teuthology.orchestra.run.smithi120.stdout:55834574852 2024-02-06T21:41:41.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 10 librados: watch_flush enter 2024-02-06T21:41:41.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 10 librados: watch_flush exit 2024-02-06T21:41:41.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 20 client.4350.objecter shutdown clearing up homeless session... 2024-02-06T21:41:41.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 10 client.4350.objecter successfully canceled tick 2024-02-06T21:41:41.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 1 -- 172.21.15.120:0/3440068232 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f029806d1f0 msgr2=0x7f029806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 1 --2- 172.21.15.120:0/3440068232 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f029806d1f0 0x7f029806f6b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009670 tx=0x7f02b0006bf0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 10 monclient: shutdown 2024-02-06T21:41:41.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:41.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 1 -- 172.21.15.120:0/3440068232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b413f830 msgr2=0x7f02b40b1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 1 --2- 172.21.15.120:0/3440068232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b413f830 0x7f02b40b1190 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f02a40097f0 tx=0x7f02a40046c0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 1 -- 172.21.15.120:0/3440068232 shutdown_connections 2024-02-06T21:41:41.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 1 --2- 172.21.15.120:0/3440068232 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f029806d1f0 0x7f029806f6b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 1 --2- 172.21.15.120:0/3440068232 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02b4142680 0x7f02b41d76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 1 --2- 172.21.15.120:0/3440068232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b413f830 0x7f02b40b1190 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 1 --2- 172.21.15.120:0/3440068232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b40b54b0 0x7f02b41d82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 1 -- 172.21.15.120:0/3440068232 >> 172.21.15.120:0/3440068232 conn(0x7f02b40b8660 msgr2=0x7f02b40a9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:41.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 1 -- 172.21.15.120:0/3440068232 shutdown_connections 2024-02-06T21:41:41.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 1 -- 172.21.15.120:0/3440068232 wait complete. 2024-02-06T21:41:41.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.090+0000 7f02bbe86640 1 librados: shutdown 2024-02-06T21:41:41.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.096+0000 7f61b159a640 1 -- 172.21.15.120:0/448305521 <== mon.0 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494022472 0 0) 0x7f61b4143200 con 0x7f61b4142680 2024-02-06T21:41:41.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.096+0000 7f61b159a640 10 monclient(hunting): _negotiate 2024-02-06T21:41:41.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.096+0000 7f61b159a640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:41.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.096+0000 7f61b159a640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:41.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.096+0000 7f61b159a640 10 monclient(hunting): my global_id is 4359 2024-02-06T21:41:41.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.097+0000 7f61b159a640 1 -- 172.21.15.120:0/448305521 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f619c003520 con 0x7f61b4142680 2024-02-06T21:41:41.105 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2024-02-06T21:41:41.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 1 -- 172.21.15.120:0/448305521 <== mon.0 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3899340093 0 0) 0x7f619c003520 con 0x7f61b4142680 2024-02-06T21:41:41.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 1 -- 172.21.15.120:0/448305521 >> v2:172.21.15.135:3300/0 conn(0x7f61b40b54b0 msgr2=0x7f61b40a9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:41.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 1 --2- 172.21.15.120:0/448305521 >> v2:172.21.15.135:3300/0 conn(0x7f61b40b54b0 0x7f61b40a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 1 -- 172.21.15.120:0/448305521 >> v1:172.21.15.135:6789/0 conn(0x7f61b413f830 legacy=0x7f61b413fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:41.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient: _finish_hunting 0 2024-02-06T21:41:41.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 1 monclient: found mon.noname-a-legacy 2024-02-06T21:41:41.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:41.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.120:6789/0 2024-02-06T21:41:41.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 1 -- 172.21.15.120:0/448305521 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b40b4800 con 0x7f61b4142680 2024-02-06T21:41:41.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient: _finish_auth 0 2024-02-06T21:41:41.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient: _check_auth_tickets 2024-02-06T21:41:41.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:41.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 1 -- 172.21.15.120:0/448305521 <== mon.0 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2420367567 0 0) 0x7f61ac0019f0 con 0x7f61b4142680 2024-02-06T21:41:41.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:41.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-02-06T21:41:41.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient: dump: 2024-02-06T21:41:41.208 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:41.209 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:41.209 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.209 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.209 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:41.209 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:41.209 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:41.209 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:41.209 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:41.209 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:41.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] but i'm connected to v1:172.21.15.120:6789/0, reconnecting 2024-02-06T21:41:41.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:41.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 1 -- 172.21.15.120:0/448305521 >> v1:172.21.15.120:6789/0 conn(0x7f61b4142680 legacy=0x7f61b4142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient: _add_conns ranks=[0,1,2] 2024-02-06T21:41:41.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 1 --2- 172.21.15.120:0/448305521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0039c0 0x7f619c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:41.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient(hunting): picked mon.a con 0x7f619c0039c0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:41.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 1 --2- 172.21.15.120:0/448305521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f619c0047c0 0x7f619c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:41.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient(hunting): picked mon.b con 0x7f619c0047c0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:41.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 1 --2- 172.21.15.120:0/448305521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c0052a0 0x7f619c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:41.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient(hunting): picked mon.c con 0x7f619c0052a0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:41.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:41.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 1 -- 172.21.15.120:0/448305521 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40ab520 con 0x7f619c0039c0 2024-02-06T21:41:41.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:41.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 1 -- 172.21.15.120:0/448305521 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c00c1f0 con 0x7f619c0047c0 2024-02-06T21:41:41.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:41.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 1 -- 172.21.15.120:0/448305521 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619c00c3b0 con 0x7f619c0052a0 2024-02-06T21:41:41.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:41.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.205+0000 7f61b159a640 10 monclient(hunting): _finish_auth 0 2024-02-06T21:41:41.211 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.206+0000 7f61b259c640 1 --2- 172.21.15.120:0/448305521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0039c0 0x7f619c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:41.211 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.207+0000 7f61b259c640 10 monclient(hunting): get_auth_request con 0x7f619c0039c0 auth_method 0 2024-02-06T21:41:41.211 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.207+0000 7f61b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:41.211 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.207+0000 7f61b259c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:41.211 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.207+0000 7f61b259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-02-06T21:41:41.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.208+0000 7f61b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:41.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.208+0000 7f61b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:41.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.208+0000 7f61b259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-02-06T21:41:41.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b259c640 10 monclient(hunting): handle_auth_done global_id 4359 payload 293 2024-02-06T21:41:41.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b259c640 1 -- 172.21.15.120:0/448305521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c0052a0 msgr2=0x7f619c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:41.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b259c640 1 --2- 172.21.15.120:0/448305521 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c0052a0 0x7f619c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b259c640 1 -- 172.21.15.120:0/448305521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f619c0047c0 msgr2=0x7f619c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:41.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b259c640 1 --2- 172.21.15.120:0/448305521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f619c0047c0 0x7f619c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b259c640 10 monclient: _finish_hunting 0 2024-02-06T21:41:41.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b259c640 1 monclient: found mon.a 2024-02-06T21:41:41.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:41.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b259c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:41.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b259c640 1 -- 172.21.15.120:0/448305521 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c003520 con 0x7f619c0039c0 2024-02-06T21:41:41.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b259c640 10 monclient: _finish_auth 0 2024-02-06T21:41:41.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b259c640 10 monclient: _check_auth_tickets 2024-02-06T21:41:41.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:41.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b2d9d640 1 -- 172.21.15.120:0/448305521 reap_dead start 2024-02-06T21:41:41.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b8d04640 5 monclient: authenticate success, global_id 4359 2024-02-06T21:41:41.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.211+0000 7f61b8d04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:41.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.214+0000 7f61b259c640 1 --2- 172.21.15.120:0/448305521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0039c0 0x7f619c003dc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f61a4006fd0 tx=0x7f619c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b90222dd94fdad server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:41.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.220+0000 7f61b159a640 1 -- 172.21.15.120:0/448305521 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4032600 con 0x7f619c0039c0 2024-02-06T21:41:41.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.220+0000 7f61b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:41.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.220+0000 7f61b159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:41.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.220+0000 7f61b159a640 10 monclient: dump: 2024-02-06T21:41:41.222 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:41.222 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:41.222 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.222 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.222 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:41.222 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:41.222 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:41.222 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:41.222 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:41.222 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:41.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.220+0000 7f61b159a640 1 -- 172.21.15.120:0/448305521 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a402e050 con 0x7f619c0039c0 2024-02-06T21:41:41.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.220+0000 7f61b8d04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:41.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.220+0000 7f61b159a640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:41.223 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.220+0000 7f61b159a640 1 -- 172.21.15.120:0/448305521 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a40329c0 con 0x7f619c0039c0 2024-02-06T21:41:41.223 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.220+0000 7f61b8d04640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:41.223 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.220+0000 7f61b8d04640 4 set_mon_vals no callback set 2024-02-06T21:41:41.223 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.220+0000 7f61b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:41.223 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.220+0000 7f61b159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:41.223 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.220+0000 7f61b159a640 10 monclient: dump: 2024-02-06T21:41:41.223 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:41.223 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:41.223 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.223 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.223 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:41.223 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:41.223 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:41.224 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:41.224 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:41.224 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:41.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.221+0000 7f61b8d04640 10 monclient: shutdown 2024-02-06T21:41:41.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.221+0000 7f61b8d04640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:41.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.221+0000 7f61b8d04640 1 -- 172.21.15.120:0/448305521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0039c0 msgr2=0x7f619c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.221+0000 7f61b8d04640 1 --2- 172.21.15.120:0/448305521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0039c0 0x7f619c003dc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f61a4006fd0 tx=0x7f619c0057f0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.221+0000 7f61b8d04640 1 -- 172.21.15.120:0/448305521 shutdown_connections 2024-02-06T21:41:41.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.221+0000 7f61b8d04640 1 --2- 172.21.15.120:0/448305521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0039c0 0x7f619c003dc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.221+0000 7f61b8d04640 1 -- 172.21.15.120:0/448305521 >> 172.21.15.120:0/448305521 conn(0x7f61b40b8660 msgr2=0x7f61b40b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:41.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.221+0000 7f61b8d04640 1 -- 172.21.15.120:0/448305521 shutdown_connections 2024-02-06T21:41:41.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.221+0000 7f61b8d04640 1 -- 172.21.15.120:0/448305521 wait complete. 2024-02-06T21:41:41.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.222+0000 7f61b8d04640 10 monclient: build_initial_monmap 2024-02-06T21:41:41.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.222+0000 7f61b8d04640 10 monclient: monmap: 2024-02-06T21:41:41.226 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:41.226 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:41.226 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:41.226 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:41.227 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:41.227 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:41.227 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:41.227 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:41.227 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:41.227 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:41.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.222+0000 7f61b8d04640 1 librados: starting msgr at 2024-02-06T21:41:41.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.222+0000 7f61b8d04640 1 librados: starting objecter 2024-02-06T21:41:41.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.223+0000 7f61b8d04640 1 Processor -- start 2024-02-06T21:41:41.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.223+0000 7f61b8d04640 1 -- start start 2024-02-06T21:41:41.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.223+0000 7f61b8d04640 1 librados: setting wanted keys 2024-02-06T21:41:41.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.223+0000 7f61b8d04640 1 librados: calling monclient init 2024-02-06T21:41:41.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.223+0000 7f61b8d04640 10 monclient: init 2024-02-06T21:41:41.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.223+0000 7f61b8d04640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:41.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.223+0000 7f61b8d04640 10 monclient: _add_conns ranks=[0,1,2] 2024-02-06T21:41:41.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.223+0000 7f61b8d04640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0039c0 0x7f61b41df280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:41.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.223+0000 7f61b8d04640 10 monclient(hunting): picked mon.noname-a con 0x7f619c0039c0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:41.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.223+0000 7f61b8d04640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61b41df7c0 0x7f61b406e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:41.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.223+0000 7f61b8d04640 10 monclient(hunting): picked mon.noname-b con 0x7f61b41df7c0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:41.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.224+0000 7f61b8d04640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61b406ed40 0x7f61b40711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:41.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.224+0000 7f61b8d04640 10 monclient(hunting): picked mon.noname-c con 0x7f61b406ed40 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:41.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.224+0000 7f61b8d04640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:41.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.224+0000 7f61b8d04640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40b1a70 con 0x7f619c0039c0 2024-02-06T21:41:41.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.224+0000 7f61b8d04640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:41.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.224+0000 7f61b8d04640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b406e370 con 0x7f61b41df7c0 2024-02-06T21:41:41.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.224+0000 7f61b8d04640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:41.228 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.224+0000 7f61b8d04640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40ac5b0 con 0x7f61b406ed40 2024-02-06T21:41:41.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.224+0000 7f61b8d04640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:41.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.224+0000 7f61b8d04640 10 monclient(hunting): authenticate will time out at 1331.561689s 2024-02-06T21:41:41.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.224+0000 7f61b259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0039c0 0x7f61b41df280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:41.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.225+0000 7f61b259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0039c0 0x7f61b41df280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60418/0 (socket says 172.21.15.120:60418) 2024-02-06T21:41:41.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.225+0000 7f61b259c640 1 -- 172.21.15.120:0/1892817682 learned_addr learned my addr 172.21.15.120:0/1892817682 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:41.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.225+0000 7f61b259c640 10 monclient(hunting): get_auth_request con 0x7f619c0039c0 auth_method 0 2024-02-06T21:41:41.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.225+0000 7f61b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:41.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.225+0000 7f61b259c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:41.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.225+0000 7f61b259c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:41.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.226+0000 7f61b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:41.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.226+0000 7f61b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:41.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.226+0000 7f61b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:41.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.229+0000 7f61b259c640 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2024-02-06T21:41:41.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.229+0000 7f61b259c640 1 -- 172.21.15.120:0/1892817682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61b406ed40 msgr2=0x7f61b40711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.229+0000 7f61b259c640 1 --2- 172.21.15.120:0/1892817682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61b406ed40 0x7f61b40711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.229+0000 7f61b259c640 1 -- 172.21.15.120:0/1892817682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61b41df7c0 msgr2=0x7f61b406e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.229+0000 7f61b259c640 1 --2- 172.21.15.120:0/1892817682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61b41df7c0 0x7f61b406e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.229+0000 7f61b259c640 10 monclient: _finish_hunting 0 2024-02-06T21:41:41.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.229+0000 7f61b259c640 1 monclient: found mon.noname-a 2024-02-06T21:41:41.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.229+0000 7f61b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:41.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.229+0000 7f61b259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:41.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.229+0000 7f61b259c640 1 -- 172.21.15.120:0/1892817682 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b40ac820 con 0x7f619c0039c0 2024-02-06T21:41:41.232 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.231+0000 7f61b259c640 1 --2- 172.21.15.120:0/1892817682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0039c0 0x7f61b41df280 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f61a4000c00 tx=0x7f61a40047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:41.233 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.231+0000 7f61a2ffd640 10 client.?.objecter ms_handle_connect 0x7f619c0039c0 2024-02-06T21:41:41.233 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.231+0000 7f61a2ffd640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:41.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.236+0000 7f61a2ffd640 1 -- 172.21.15.120:0/1892817682 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a403c880 con 0x7f619c0039c0 2024-02-06T21:41:41.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.236+0000 7f61a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:41.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.236+0000 7f61a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:41.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.236+0000 7f61a2ffd640 10 monclient: dump: 2024-02-06T21:41:41.238 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:41.238 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:41.238 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.238 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.239 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:41.239 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:41.239 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:41.239 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:41.239 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:41.239 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:41.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.236+0000 7f61a2ffd640 10 monclient: _finish_auth 0 2024-02-06T21:41:41.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.236+0000 7f61a2ffd640 10 monclient: _check_auth_tickets 2024-02-06T21:41:41.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.236+0000 7f61a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:41.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.236+0000 7f61a2ffd640 1 -- 172.21.15.120:0/1892817682 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a403c9e0 con 0x7f619c0039c0 2024-02-06T21:41:41.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.236+0000 7f61b8d04640 5 monclient: authenticate success, global_id 4362 2024-02-06T21:41:41.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.236+0000 7f61b8d04640 10 monclient: _renew_subs 2024-02-06T21:41:41.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.236+0000 7f61b8d04640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:41.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.236+0000 7f61b8d04640 1 -- 172.21.15.120:0/1892817682 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b41dfc60 con 0x7f619c0039c0 2024-02-06T21:41:41.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.236+0000 7f61a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:41.241 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.236+0000 7f61b0d99640 4 set_mon_vals no callback set 2024-02-06T21:41:41.241 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.236+0000 7f61b8d04640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:41.241 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.237+0000 7f61a2ffd640 1 -- 172.21.15.120:0/1892817682 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a403cd90 con 0x7f619c0039c0 2024-02-06T21:41:41.241 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.237+0000 7f61a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:41.241 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.237+0000 7f61a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:41.241 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.237+0000 7f61a2ffd640 10 monclient: dump: 2024-02-06T21:41:41.241 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:41.241 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:41.241 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.241 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.241 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:41.241 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:41.242 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:41.242 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:41.242 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:41.242 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:41.242 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.237+0000 7f61b8d04640 10 monclient: _renew_subs 2024-02-06T21:41:41.242 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.237+0000 7f61b8d04640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:41.242 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.237+0000 7f61b8d04640 1 -- 172.21.15.120:0/1892817682 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b406f3b0 con 0x7f619c0039c0 2024-02-06T21:41:41.242 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.237+0000 7f61b8d04640 1 librados: init done 2024-02-06T21:41:41.242 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.238+0000 7f61b8d04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:41.242 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.238+0000 7f61b8d04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:41.242 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.238+0000 7f61b8d04640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:41.242 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.238+0000 7f61b8d04640 1 -- 172.21.15.120:0/1892817682 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b404ef90 con 0x7f619c0039c0 2024-02-06T21:41:41.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.264+0000 7ff9e3fff640 1 -- 172.21.15.120:0/2116432464 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff9ec07bc70 con 0x7ff9fc0a9c80 2024-02-06T21:41:41.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.264+0000 7ff9e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-02-06T21:41:41.266 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.264+0000 7ff9e3fff640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:41.266 INFO:teuthology.orchestra.run.smithi120.stdout:73014444039 2024-02-06T21:41:41.267 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.265+0000 7f61a2ffd640 1 -- 172.21.15.120:0/1892817682 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f61a4039040 con 0x7f619c0039c0 2024-02-06T21:41:41.267 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.266+0000 7f61a2ffd640 1 --2- 172.21.15.120:0/1892817682 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f619406d1a0 0x7f619406f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:41.268 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.266+0000 7f61b1d9b640 1 --2- 172.21.15.120:0/1892817682 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f619406d1a0 0x7f619406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:41.268 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.266+0000 7f61b1d9b640 10 monclient: get_auth_request con 0x7f619406d1a0 auth_method 0 2024-02-06T21:41:41.268 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7f61a2ffd640 10 client.4362.objecter ms_handle_connect 0x7f619406d1a0 2024-02-06T21:41:41.268 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7f61b1d9b640 1 --2- 172.21.15.120:0/1892817682 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f619406d1a0 0x7f619406f660 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f61b40b2080 tx=0x7f61a8007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:41.269 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 10 librados: watch_flush enter 2024-02-06T21:41:41.269 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 10 librados: watch_flush exit 2024-02-06T21:41:41.269 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 20 client.4319.objecter shutdown clearing up homeless session... 2024-02-06T21:41:41.269 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 10 client.4319.objecter successfully canceled tick 2024-02-06T21:41:41.270 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 1 -- 172.21.15.120:0/2116432464 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff9e806d1f0 msgr2=0x7ff9e806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.270 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 1 --2- 172.21.15.120:0/2116432464 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff9e806d1f0 0x7ff9e806f6b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc13abf0 tx=0x7ff9f0006bf0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.270 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 10 monclient: shutdown 2024-02-06T21:41:41.270 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:41.270 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 1 -- 172.21.15.120:0/2116432464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9fc0a9c80 msgr2=0x7ff9fc13a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.270 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 1 --2- 172.21.15.120:0/2116432464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9fc0a9c80 0x7ff9fc13a3f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009e50 tx=0x7ff9ec004a40 comp rx=0 tx=0).stop 2024-02-06T21:41:41.270 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 1 -- 172.21.15.120:0/2116432464 shutdown_connections 2024-02-06T21:41:41.270 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 1 --2- 172.21.15.120:0/2116432464 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff9e806d1f0 0x7ff9e806f6b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.270 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 1 --2- 172.21.15.120:0/2116432464 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9fc142bb0 0x7ff9fc1354e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.270 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 1 --2- 172.21.15.120:0/2116432464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9fc13f810 0x7ff9fc136730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.270 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 1 --2- 172.21.15.120:0/2116432464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9fc0a9c80 0x7ff9fc13a3f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.270 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.267+0000 7ffa01f23640 1 -- 172.21.15.120:0/2116432464 >> 172.21.15.120:0/2116432464 conn(0x7ff9fc0b4ba0 msgr2=0x7ff9fc06d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:41.271 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.269+0000 7ffa01f23640 1 -- 172.21.15.120:0/2116432464 shutdown_connections 2024-02-06T21:41:41.271 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.269+0000 7ffa01f23640 1 -- 172.21.15.120:0/2116432464 wait complete. 2024-02-06T21:41:41.271 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.269+0000 7ffa01f23640 1 librados: shutdown 2024-02-06T21:41:41.273 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.271+0000 7f61a2ffd640 1 -- 172.21.15.120:0/1892817682 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f61a407a660 con 0x7f619c0039c0 2024-02-06T21:41:41.273 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.271+0000 7f61a2ffd640 10 client.4362.objecter ms_dispatch 0x7f61b4000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:41.273 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.271+0000 7f61a2ffd640 3 client.4362.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:41.273 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.271+0000 7f61a2ffd640 3 client.4362.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:41.273 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.272+0000 7f61a2ffd640 20 client.4362.objecter dump_active .. 0 homeless 2024-02-06T21:41:41.276 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.275+0000 7fcb6ed1c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-02-06T21:41:41.276 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.275+0000 7fcb6ed1c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-02-06T21:41:41.276 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.275+0000 7fcb6ed1c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:41.277 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.275+0000 7fcb6ed1c640 1 -- 172.21.15.120:0/192166374 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fcb68134360 con 0x7fcb680b54b0 2024-02-06T21:41:41.284 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444039 for osd.3 2024-02-06T21:41:41.287 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.285+0000 7f2bd65c4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-02-06T21:41:41.287 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.285+0000 7f2bd65c4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-02-06T21:41:41.287 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.285+0000 7f2bd65c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:41.287 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.285+0000 7f2bd65c4640 1 -- 172.21.15.120:0/471259868 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f2bd0068010 con 0x7f2bd00709d0 2024-02-06T21:41:41.293 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.292+0000 7f3a4bfff640 1 -- 172.21.15.120:0/2492854313 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3a5804d450 con 0x7f3a6413ff40 2024-02-06T21:41:41.294 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.292+0000 7f3a4bfff640 1 --2- 172.21.15.120:0/2492854313 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f3a5006d1a0 0x7f3a5006f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:41.294 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.293+0000 7f3a637fe640 1 --2- 172.21.15.120:0/2492854313 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f3a5006d1a0 0x7f3a5006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:41.294 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.293+0000 7f3a637fe640 10 monclient: get_auth_request con 0x7f3a5006d1a0 auth_method 0 2024-02-06T21:41:41.294 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.293+0000 7f3a637fe640 1 --2- 172.21.15.120:0/2492854313 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f3a5006d1a0 0x7f3a5006f660 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009650 tx=0x7f3a54006ba0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:41.294 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.293+0000 7f3a4bfff640 10 client.4366.objecter ms_handle_connect 0x7f3a5006d1a0 2024-02-06T21:41:41.298 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.296+0000 7f3a4bfff640 1 -- 172.21.15.120:0/2492854313 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3a5802d040 con 0x7f3a6413ff40 2024-02-06T21:41:41.298 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.296+0000 7f3a4bfff640 10 client.4366.objecter ms_dispatch 0x7f3a64000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:41.298 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.296+0000 7f3a4bfff640 3 client.4366.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:41.298 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.296+0000 7f3a4bfff640 3 client.4366.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:41.298 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.296+0000 7f3a4bfff640 20 client.4366.objecter dump_active .. 0 homeless 2024-02-06T21:41:41.311 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.309+0000 7f3a4bfff640 1 -- 172.21.15.120:0/2492854313 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3a580777d0 con 0x7f3a6413ff40 2024-02-06T21:41:41.311 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.309+0000 7f3a4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:41.311 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.309+0000 7f3a4bfff640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:41.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.367+0000 7fcb657fa640 1 -- 172.21.15.120:0/192166374 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcb68134360 con 0x7fcb680b54b0 2024-02-06T21:41:41.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.367+0000 7fcb657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-02-06T21:41:41.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.367+0000 7fcb657fa640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:41.369 INFO:teuthology.orchestra.run.smithi120.stdout:73014444039 2024-02-06T21:41:41.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 10 librados: watch_flush enter 2024-02-06T21:41:41.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 10 librados: watch_flush exit 2024-02-06T21:41:41.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 20 client.4363.objecter shutdown clearing up homeless session... 2024-02-06T21:41:41.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 10 client.4363.objecter successfully canceled tick 2024-02-06T21:41:41.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 1 -- 172.21.15.120:0/192166374 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fcb4c06d220 msgr2=0x7fcb4c06f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 1 --2- 172.21.15.120:0/192166374 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fcb4c06d220 0x7fcb4c06f6e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcb6813d390 tx=0x7fcb54007990 comp rx=0 tx=0).stop 2024-02-06T21:41:41.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 10 monclient: shutdown 2024-02-06T21:41:41.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:41.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 1 -- 172.21.15.120:0/192166374 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb680b54b0 msgr2=0x7fcb6814c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 1 --2- 172.21.15.120:0/192166374 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb680b54b0 0x7fcb6814c440 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c000c00 tx=0x7fcb5c0046c0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 1 -- 172.21.15.120:0/192166374 shutdown_connections 2024-02-06T21:41:41.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 1 --2- 172.21.15.120:0/192166374 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fcb4c06d220 0x7fcb4c06f6e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 1 --2- 172.21.15.120:0/192166374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb68142680 0x7fcb68137fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 1 --2- 172.21.15.120:0/192166374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb6813f830 0x7fcb68134620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 1 --2- 172.21.15.120:0/192166374 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb680b54b0 0x7fcb6814c440 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 1 -- 172.21.15.120:0/192166374 >> 172.21.15.120:0/192166374 conn(0x7fcb680b8660 msgr2=0x7fcb680744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:41.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 1 -- 172.21.15.120:0/192166374 shutdown_connections 2024-02-06T21:41:41.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 1 -- 172.21.15.120:0/192166374 wait complete. 2024-02-06T21:41:41.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.369+0000 7fcb6ed1c640 1 librados: shutdown 2024-02-06T21:41:41.382 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444039 for osd.6 2024-02-06T21:41:41.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.510+0000 7f3a6a13b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-02-06T21:41:41.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.510+0000 7f3a6a13b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-02-06T21:41:41.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.510+0000 7f3a6a13b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:41.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.510+0000 7f3a6a13b640 1 -- 172.21.15.120:0/2492854313 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f3a6404ef90 con 0x7f3a6413ff40 2024-02-06T21:41:41.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.514+0000 7f3a4bfff640 1 -- 172.21.15.120:0/2492854313 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3a5807ae90 con 0x7f3a6413ff40 2024-02-06T21:41:41.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.514+0000 7f3a4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-02-06T21:41:41.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.514+0000 7f3a4bfff640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:41.515 INFO:teuthology.orchestra.run.smithi120.stdout:55834574853 2024-02-06T21:41:41.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 10 librados: watch_flush enter 2024-02-06T21:41:41.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 10 librados: watch_flush exit 2024-02-06T21:41:41.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 20 client.4366.objecter shutdown clearing up homeless session... 2024-02-06T21:41:41.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 10 client.4366.objecter successfully canceled tick 2024-02-06T21:41:41.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 1 -- 172.21.15.120:0/2492854313 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f3a5006d1a0 msgr2=0x7f3a5006f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 1 --2- 172.21.15.120:0/2492854313 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f3a5006d1a0 0x7f3a5006f660 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009650 tx=0x7f3a54006ba0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 10 monclient: shutdown 2024-02-06T21:41:41.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:41.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 1 -- 172.21.15.120:0/2492854313 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a6413ff40 msgr2=0x7f3a64137a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 1 --2- 172.21.15.120:0/2492854313 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a6413ff40 0x7f3a64137a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3a58000c00 tx=0x7f3a580046c0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 1 -- 172.21.15.120:0/2492854313 shutdown_connections 2024-02-06T21:41:41.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 1 --2- 172.21.15.120:0/2492854313 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f3a5006d1a0 0x7f3a5006f660 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 1 --2- 172.21.15.120:0/2492854313 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a6413ff40 0x7f3a64137a00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 1 --2- 172.21.15.120:0/2492854313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a64142680 0x7f3a64135570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 1 --2- 172.21.15.120:0/2492854313 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a640b54b0 0x7f3a640b7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 1 -- 172.21.15.120:0/2492854313 >> 172.21.15.120:0/2492854313 conn(0x7f3a640b8660 msgr2=0x7f3a641349c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:41.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 1 -- 172.21.15.120:0/2492854313 shutdown_connections 2024-02-06T21:41:41.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 1 -- 172.21.15.120:0/2492854313 wait complete. 2024-02-06T21:41:41.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.516+0000 7f3a6a13b640 1 librados: shutdown 2024-02-06T21:41:41.529 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2024-02-06T21:41:41.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.560+0000 7f61a2ffd640 1 -- 172.21.15.120:0/1892817682 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f61a4076900 con 0x7f619c0039c0 2024-02-06T21:41:41.562 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.560+0000 7f61a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:41.562 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.560+0000 7f61a2ffd640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:41.612 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.611+0000 7f2bccff9640 1 -- 172.21.15.120:0/471259868 <== mon.0 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2bc0014090 con 0x7f2bd00709d0 2024-02-06T21:41:41.612 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.611+0000 7f2bccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-02-06T21:41:41.612 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.611+0000 7f2bccff9640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:41.613 INFO:teuthology.orchestra.run.smithi120.stdout:73014444039 2024-02-06T21:41:41.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.612+0000 7f2bd65c4640 10 librados: watch_flush enter 2024-02-06T21:41:41.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.612+0000 7f2bd65c4640 10 librados: watch_flush exit 2024-02-06T21:41:41.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.612+0000 7f2bd65c4640 20 client.4356.objecter shutdown clearing up homeless session... 2024-02-06T21:41:41.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.612+0000 7f2bd65c4640 10 client.4356.objecter successfully canceled tick 2024-02-06T21:41:41.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.613+0000 7f2bd65c4640 1 -- 172.21.15.120:0/471259868 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f2bbc06d220 msgr2=0x7f2bbc06f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.613+0000 7f2bd65c4640 1 --2- 172.21.15.120:0/471259868 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f2bbc06d220 0x7f2bbc06f6e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4036000 tx=0x7f2bc4030000 comp rx=0 tx=0).stop 2024-02-06T21:41:41.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.613+0000 7f2bd65c4640 10 monclient: shutdown 2024-02-06T21:41:41.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.613+0000 7f2bd65c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:41.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.613+0000 7f2bd65c4640 1 -- 172.21.15.120:0/471259868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bd00709d0 msgr2=0x7f2bd01443f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.613+0000 7f2bd65c4640 1 --2- 172.21.15.120:0/471259868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bd00709d0 0x7f2bd01443f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2bc000b4d0 tx=0x7f2bc000b990 comp rx=0 tx=0).stop 2024-02-06T21:41:41.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.613+0000 7f2bd65c4640 1 -- 172.21.15.120:0/471259868 shutdown_connections 2024-02-06T21:41:41.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.613+0000 7f2bd65c4640 1 --2- 172.21.15.120:0/471259868 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f2bbc06d220 0x7f2bbc06f6e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.613+0000 7f2bd65c4640 1 --2- 172.21.15.120:0/471259868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2bd013d2a0 0x7f2bd01d9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.616 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.613+0000 7f2bd65c4640 1 --2- 172.21.15.120:0/471259868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bd0070db0 0x7f2bd00a9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.616 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.613+0000 7f2bd65c4640 1 --2- 172.21.15.120:0/471259868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bd00709d0 0x7f2bd01443f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.616 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.613+0000 7f2bd65c4640 1 -- 172.21.15.120:0/471259868 >> 172.21.15.120:0/471259868 conn(0x7f2bd00b0bc0 msgr2=0x7f2bd0140600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:41.616 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.613+0000 7f2bd65c4640 1 -- 172.21.15.120:0/471259868 shutdown_connections 2024-02-06T21:41:41.616 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.613+0000 7f2bd65c4640 1 -- 172.21.15.120:0/471259868 wait complete. 2024-02-06T21:41:41.616 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.613+0000 7f2bd65c4640 1 librados: shutdown 2024-02-06T21:41:41.633 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444039 for osd.2 2024-02-06T21:41:41.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.760+0000 7f61b8d04640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-02-06T21:41:41.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.760+0000 7f61b8d04640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-02-06T21:41:41.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.760+0000 7f61b8d04640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:41.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.760+0000 7f61b8d04640 1 -- 172.21.15.120:0/1892817682 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f61b413a4c0 con 0x7f619c0039c0 2024-02-06T21:41:41.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.763+0000 7f61a2ffd640 1 -- 172.21.15.120:0/1892817682 <== mon.0 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f61a404d020 con 0x7f619c0039c0 2024-02-06T21:41:41.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.763+0000 7f61a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-02-06T21:41:41.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.763+0000 7f61a2ffd640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:41.765 INFO:teuthology.orchestra.run.smithi120.stdout:73014444043 2024-02-06T21:41:41.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 10 librados: watch_flush enter 2024-02-06T21:41:41.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 10 librados: watch_flush exit 2024-02-06T21:41:41.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 20 client.4362.objecter shutdown clearing up homeless session... 2024-02-06T21:41:41.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 10 client.4362.objecter successfully canceled tick 2024-02-06T21:41:41.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 1 -- 172.21.15.120:0/1892817682 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f619406d1a0 msgr2=0x7f619406f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 1 --2- 172.21.15.120:0/1892817682 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f619406d1a0 0x7f619406f660 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f61b40b2080 tx=0x7f61a8007990 comp rx=0 tx=0).stop 2024-02-06T21:41:41.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 10 monclient: shutdown 2024-02-06T21:41:41.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:41.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 1 -- 172.21.15.120:0/1892817682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0039c0 msgr2=0x7f61b41df280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 1 --2- 172.21.15.120:0/1892817682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0039c0 0x7f61b41df280 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f61a4000c00 tx=0x7f61a40047d0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 1 -- 172.21.15.120:0/1892817682 shutdown_connections 2024-02-06T21:41:41.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 1 --2- 172.21.15.120:0/1892817682 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f619406d1a0 0x7f619406f660 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 1 --2- 172.21.15.120:0/1892817682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61b41df7c0 0x7f61b406e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 1 --2- 172.21.15.120:0/1892817682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61b406ed40 0x7f61b40711d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 1 --2- 172.21.15.120:0/1892817682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c0039c0 0x7f61b41df280 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.765+0000 7f61b8d04640 1 -- 172.21.15.120:0/1892817682 >> 172.21.15.120:0/1892817682 conn(0x7f61b40b8660 msgr2=0x7f61b413fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:41.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.766+0000 7f61b8d04640 1 -- 172.21.15.120:0/1892817682 shutdown_connections 2024-02-06T21:41:41.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.766+0000 7f61b8d04640 1 -- 172.21.15.120:0/1892817682 wait complete. 2024-02-06T21:41:41.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.766+0000 7f61b8d04640 1 librados: shutdown 2024-02-06T21:41:41.778 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444042 got 73014444043 for osd.5 2024-02-06T21:41:41.778 DEBUG:teuthology.parallel:result is None 2024-02-06T21:41:41.835 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-02-06T21:41:41.937 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.935+0000 7f8021f04640 10 monclient: get_monmap_and_config 2024-02-06T21:41:41.937 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.935+0000 7f8021f04640 10 monclient: build_initial_monmap 2024-02-06T21:41:41.937 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.935+0000 7f8021f04640 10 monclient: monmap: 2024-02-06T21:41:41.937 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:41.937 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:41.937 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:41.936821+0000 2024-02-06T21:41:41.937 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:41.936821+0000 2024-02-06T21:41:41.937 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:41.937 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:41.937 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:41.937 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:41.938 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:41.938 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:41.938 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:41.938 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:41.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.936+0000 7f8021f04640 1 Processor -- start 2024-02-06T21:41:41.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.936+0000 7f8021f04640 1 -- start start 2024-02-06T21:41:41.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.936+0000 7f8021f04640 10 monclient: init 2024-02-06T21:41:41.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.936+0000 7f8021f04640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:41.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.936+0000 7f8021f04640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-02-06T21:41:41.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.936+0000 7f8021f04640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f801c0a9c80 0x7f801c0aa060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:41.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.936+0000 7f8021f04640 10 monclient(hunting): picked mon.noname-c con 0x7f801c0a9c80 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:41.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.936+0000 7f8021f04640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c142bb0 0x7f801c142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:41.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f8021f04640 10 monclient(hunting): picked mon.noname-b con 0x7f801c142bb0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:41.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f8021f04640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f801c13f810 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:41.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f8021f04640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c0aa5a0 con 0x7f801c13f810 2024-02-06T21:41:41.940 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f8021f04640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:41.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f8021f04640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f801c143510 con 0x7f801c0a9c80 2024-02-06T21:41:41.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f8021f04640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:41.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f8021f04640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801c142720 con 0x7f801c142bb0 2024-02-06T21:41:41.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f8021f04640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:41.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f8021f04640 10 monclient(hunting): authenticate will time out at 1332.274580s 2024-02-06T21:41:41.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f801affd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c142bb0 0x7f801c142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:41.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f801affd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c142bb0 0x7f801c142fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57148/0 (socket says 172.21.15.120:57148) 2024-02-06T21:41:41.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f801affd640 1 -- 172.21.15.120:0/2903039100 learned_addr learned my addr 172.21.15.120:0/2903039100 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:41.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f801affd640 10 monclient(hunting): get_auth_request con 0x7f801c142bb0 auth_method 0 2024-02-06T21:41:41.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f801affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:41.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f801affd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:41.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.937+0000 7f801affd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:41.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.938+0000 7f801affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:41.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.938+0000 7f801affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:41.941 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.938+0000 7f801affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:41.942 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.940+0000 7f801affd640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2024-02-06T21:41:41.942 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.940+0000 7f801affd640 1 -- 172.21.15.120:0/2903039100 >> v2:172.21.15.135:3300/0 conn(0x7f801c0a9c80 msgr2=0x7f801c0aa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:41.942 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.940+0000 7f801affd640 1 --2- 172.21.15.120:0/2903039100 >> v2:172.21.15.135:3300/0 conn(0x7f801c0a9c80 0x7f801c0aa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.942 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.940+0000 7f801affd640 1 -- 172.21.15.120:0/2903039100 >> v1:172.21.15.135:6789/0 conn(0x7f801c13f810 legacy=0x7f801c13fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:41.942 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.940+0000 7f801affd640 10 monclient: _finish_hunting 0 2024-02-06T21:41:41.942 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.940+0000 7f801affd640 1 monclient: found mon.noname-b 2024-02-06T21:41:41.943 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.940+0000 7f801affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:41.943 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.940+0000 7f801affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:41.943 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.940+0000 7f801affd640 1 -- 172.21.15.120:0/2903039100 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801c0ab120 con 0x7f801c142bb0 2024-02-06T21:41:41.943 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.942+0000 7f801affd640 1 --2- 172.21.15.120:0/2903039100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c142bb0 0x7f801c142fd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8010009870 tx=0x7f801002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdc40ec84e404303 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:41.947 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f801a7fc640 1 -- 172.21.15.120:0/2903039100 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010031640 con 0x7f801c142bb0 2024-02-06T21:41:41.947 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f801a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:41.947 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f801a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:41.947 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f801a7fc640 10 monclient: dump: 2024-02-06T21:41:41.947 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:41.947 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:41.948 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.948 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.948 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:41.948 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:41.948 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:41.948 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:41.948 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:41.948 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:41.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f801a7fc640 10 monclient: _finish_auth 0 2024-02-06T21:41:41.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f801a7fc640 10 monclient: _check_auth_tickets 2024-02-06T21:41:41.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f801a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:41.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f801a7fc640 1 -- 172.21.15.120:0/2903039100 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80100317a0 con 0x7f801c142bb0 2024-02-06T21:41:41.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f801a7fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:41.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f801a7fc640 1 -- 172.21.15.120:0/2903039100 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010031b30 con 0x7f801c142bb0 2024-02-06T21:41:41.949 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f801a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:41.949 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f801a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:41.949 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f801a7fc640 10 monclient: dump: 2024-02-06T21:41:41.949 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:41.949 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:41.949 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.949 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.949 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:41.949 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:41.949 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:41.949 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:41.949 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:41.949 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:41.950 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f8021f04640 5 monclient: authenticate success, global_id 4325 2024-02-06T21:41:41.950 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f8021f04640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:41.950 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.945+0000 7f8021f04640 4 set_mon_vals no callback set 2024-02-06T21:41:41.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.946+0000 7f8021f04640 10 monclient: shutdown 2024-02-06T21:41:41.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.946+0000 7f8021f04640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:41.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.946+0000 7f8021f04640 1 -- 172.21.15.120:0/2903039100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c142bb0 msgr2=0x7f801c142fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.946+0000 7f8021f04640 1 --2- 172.21.15.120:0/2903039100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c142bb0 0x7f801c142fd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8010009870 tx=0x7f801002b8f0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.946+0000 7f8021f04640 1 -- 172.21.15.120:0/2903039100 shutdown_connections 2024-02-06T21:41:41.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.946+0000 7f8021f04640 1 --2- 172.21.15.120:0/2903039100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c142bb0 0x7f801c142fd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.946+0000 7f8021f04640 1 --2- 172.21.15.120:0/2903039100 >> v2:172.21.15.135:3300/0 conn(0x7f801c0a9c80 0x7f801c0aa060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.946+0000 7f8021f04640 1 -- 172.21.15.120:0/2903039100 >> 172.21.15.120:0/2903039100 conn(0x7f801c0b4ba0 msgr2=0x7f801c0b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:41.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.946+0000 7f8021f04640 1 -- 172.21.15.120:0/2903039100 shutdown_connections 2024-02-06T21:41:41.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.946+0000 7f8021f04640 1 -- 172.21.15.120:0/2903039100 wait complete. 2024-02-06T21:41:41.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.947+0000 7f8021f04640 10 monclient: build_initial_monmap 2024-02-06T21:41:41.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.947+0000 7f8021f04640 10 monclient: monmap: 2024-02-06T21:41:41.954 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:41.954 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:41.954 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:41.954 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:41.954 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:41.954 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:41.954 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:41.954 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:41.954 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:41.954 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:41.954 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.947+0000 7f8021f04640 1 librados: starting msgr at 2024-02-06T21:41:41.954 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.947+0000 7f8021f04640 1 librados: starting objecter 2024-02-06T21:41:41.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.947+0000 7f8021f04640 1 Processor -- start 2024-02-06T21:41:41.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.947+0000 7f8021f04640 1 -- start start 2024-02-06T21:41:41.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.947+0000 7f8021f04640 1 librados: setting wanted keys 2024-02-06T21:41:41.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.947+0000 7f8021f04640 1 librados: calling monclient init 2024-02-06T21:41:41.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.947+0000 7f8021f04640 10 monclient: init 2024-02-06T21:41:41.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:41.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 10 monclient: _add_conns ranks=[0,1,2] 2024-02-06T21:41:41.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f801c0a9c80 0x7f801c1417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:41.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 10 monclient(hunting): picked mon.noname-a con 0x7f801c0a9c80 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:41.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f801c13f810 0x7f801c139730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:41.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 10 monclient(hunting): picked mon.noname-b con 0x7f801c13f810 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:41.955 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c142bb0 0x7f801c13a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:41.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 10 monclient(hunting): picked mon.noname-c con 0x7f801c142bb0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:41.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:41.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c1440e0 con 0x7f801c0a9c80 2024-02-06T21:41:41.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:41.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c144350 con 0x7f801c13f810 2024-02-06T21:41:41.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:41.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801c139c70 con 0x7f801c142bb0 2024-02-06T21:41:41.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:41.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f8021f04640 10 monclient(hunting): authenticate will time out at 1332.285987s 2024-02-06T21:41:41.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.948+0000 7f801bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c142bb0 0x7f801c13a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:41.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.949+0000 7f801bfff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c142bb0 0x7f801c13a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57154/0 (socket says 172.21.15.120:57154) 2024-02-06T21:41:41.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.949+0000 7f801bfff640 1 -- 172.21.15.120:0/2504394258 learned_addr learned my addr 172.21.15.120:0/2504394258 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:41.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.949+0000 7f801bfff640 10 monclient(hunting): get_auth_request con 0x7f801c142bb0 auth_method 0 2024-02-06T21:41:41.956 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.949+0000 7f801bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:41.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.949+0000 7f801bfff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:41.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.949+0000 7f801bfff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:41.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.949+0000 7f801affd640 1 --2- 172.21.15.120:0/2504394258 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f801c13f810 0x7f801c139730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:41.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.949+0000 7f801bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:41.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.949+0000 7f801bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:41.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.949+0000 7f801bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:41.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.950+0000 7f801affd640 10 monclient(hunting): get_auth_request con 0x7f801c13f810 auth_method 0 2024-02-06T21:41:41.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.950+0000 7f801affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:41.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.950+0000 7f801affd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:41.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.950+0000 7f801affd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:41.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.951+0000 7f801affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:41.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.951+0000 7f801affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:41.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.951+0000 7f801affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:41.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.952+0000 7f801bfff640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2024-02-06T21:41:41.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.952+0000 7f801bfff640 1 -- 172.21.15.120:0/2504394258 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f801c13f810 msgr2=0x7f801c139730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.952+0000 7f801bfff640 1 --2- 172.21.15.120:0/2504394258 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f801c13f810 0x7f801c139730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.952+0000 7f801bfff640 1 -- 172.21.15.120:0/2504394258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f801c0a9c80 msgr2=0x7f801c1417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:41.958 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.952+0000 7f801bfff640 1 --2- 172.21.15.120:0/2504394258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f801c0a9c80 0x7f801c1417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:41.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.952+0000 7f801bfff640 10 monclient: _finish_hunting 0 2024-02-06T21:41:41.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.952+0000 7f801bfff640 1 monclient: found mon.noname-c 2024-02-06T21:41:41.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.952+0000 7f801bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:41.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.952+0000 7f801bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:41:41.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.952+0000 7f801bfff640 1 -- 172.21.15.120:0/2504394258 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80100094d0 con 0x7f801c142bb0 2024-02-06T21:41:41.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.954+0000 7f801bfff640 1 --2- 172.21.15.120:0/2504394258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c142bb0 0x7f801c13a490 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f801400c780 tx=0x7f801400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:41.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.954+0000 7f8003fff640 10 client.?.objecter ms_handle_connect 0x7f801c142bb0 2024-02-06T21:41:41.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.954+0000 7f8003fff640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:41.961 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8003fff640 1 -- 172.21.15.120:0/2504394258 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014003d80 con 0x7f801c142bb0 2024-02-06T21:41:41.961 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:41.961 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8003fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:41.961 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8003fff640 10 monclient: dump: 2024-02-06T21:41:41.962 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:41.962 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:41.962 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.962 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.962 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:41.962 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:41.962 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:41.962 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:41.962 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:41.962 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:41.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8003fff640 10 monclient: _finish_auth 0 2024-02-06T21:41:41.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8003fff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:41.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8003fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:41.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8003fff640 1 -- 172.21.15.120:0/2504394258 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801400ce90 con 0x7f801c142bb0 2024-02-06T21:41:41.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8021f04640 5 monclient: authenticate success, global_id 4328 2024-02-06T21:41:41.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8021f04640 10 monclient: _renew_subs 2024-02-06T21:41:41.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8021f04640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:41.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8021f04640 1 -- 172.21.15.120:0/2504394258 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801c1382c0 con 0x7f801c142bb0 2024-02-06T21:41:41.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8003fff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:41.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8003fff640 1 -- 172.21.15.120:0/2504394258 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80140052b0 con 0x7f801c142bb0 2024-02-06T21:41:41.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:41.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8003fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:41.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8003fff640 10 monclient: dump: 2024-02-06T21:41:41.963 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:41.963 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:41.963 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.963 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:41.964 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:41.964 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:41.964 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:41.964 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:41.964 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:41.964 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:41.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8019ffb640 4 set_mon_vals no callback set 2024-02-06T21:41:41.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8021f04640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:41.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8021f04640 10 monclient: _renew_subs 2024-02-06T21:41:41.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8021f04640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:41.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8021f04640 1 -- 172.21.15.120:0/2504394258 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801c13abb0 con 0x7f801c142bb0 2024-02-06T21:41:41.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.959+0000 7f8021f04640 1 librados: init done 2024-02-06T21:41:41.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.960+0000 7f8021f04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:41.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.960+0000 7f8021f04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:41.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.960+0000 7f8021f04640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:41.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.960+0000 7f8021f04640 1 -- 172.21.15.120:0/2504394258 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801c13a0d0 con 0x7f801c142bb0 2024-02-06T21:41:41.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.982+0000 7f8003fff640 1 -- 172.21.15.120:0/2504394258 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8014005410 con 0x7f801c142bb0 2024-02-06T21:41:41.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.982+0000 7f8003fff640 1 --2- 172.21.15.120:0/2504394258 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f800806d1a0 0x7f800806f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:41.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.982+0000 7f801b7fe640 1 --2- 172.21.15.120:0/2504394258 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f800806d1a0 0x7f800806f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:41.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.982+0000 7f801b7fe640 10 monclient: get_auth_request con 0x7f800806d1a0 auth_method 0 2024-02-06T21:41:41.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.983+0000 7f801b7fe640 1 --2- 172.21.15.120:0/2504394258 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f800806d1a0 0x7f800806f660 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f800c009710 tx=0x7f800c006c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:41.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:41.983+0000 7f8003fff640 10 client.4328.objecter ms_handle_connect 0x7f800806d1a0 2024-02-06T21:41:42.045 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-02-06T21:41:42.106 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-02-06T21:41:42.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.146+0000 7fbda4806640 10 monclient: get_monmap_and_config 2024-02-06T21:41:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.146+0000 7fbda4806640 10 monclient: build_initial_monmap 2024-02-06T21:41:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.146+0000 7fbda4806640 10 monclient: monmap: 2024-02-06T21:41:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:42.147622+0000 2024-02-06T21:41:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:42.147622+0000 2024-02-06T21:41:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:42.149 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:42.149 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.146+0000 7fbda4806640 1 Processor -- start 2024-02-06T21:41:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.146+0000 7fbda4806640 1 -- start start 2024-02-06T21:41:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.146+0000 7fbda4806640 10 monclient: init 2024-02-06T21:41:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-02-06T21:41:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbda00b54b0 0x7fbda00a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 10 monclient(hunting): picked mon.noname-c con 0x7fbda00b54b0 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbda013f830 0x7fbda013fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 10 monclient(hunting): picked mon.noname-b con 0x7fbda013f830 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbda0142680 0x7fbda0142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 10 monclient(hunting): picked mon.noname-a con 0x7fbda0142680 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.152 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda00ab520 con 0x7fbda0142680 2024-02-06T21:41:42.152 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.152 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda00b5890 con 0x7fbda00b54b0 2024-02-06T21:41:42.152 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.152 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00aa450 con 0x7fbda013f830 2024-02-06T21:41:42.152 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:42.152 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.147+0000 7fbda4806640 10 monclient(hunting): authenticate will time out at 1332.485288s 2024-02-06T21:41:42.152 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.148+0000 7fbd9ed9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbda0142680 0x7fbda0142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.152 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.148+0000 7fbd9e59c640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbda00b54b0 0x7fbda00a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.152 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.148+0000 7fbd9e59c640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbda00b54b0 0x7fbda00a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38918/0 (socket says 172.21.15.120:38918) 2024-02-06T21:41:42.152 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.148+0000 7fbd9e59c640 1 -- 172.21.15.120:0/1366153902 learned_addr learned my addr 172.21.15.120:0/1366153902 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:42.152 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.148+0000 7fbd9e59c640 10 monclient(hunting): get_auth_request con 0x7fbda00b54b0 auth_method 0 2024-02-06T21:41:42.152 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.148+0000 7fbd9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:42.152 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.148+0000 7fbd9e59c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:42.153 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.148+0000 7fbd9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:42.153 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.148+0000 7fbd9ed9d640 10 monclient(hunting): get_auth_request con 0x7fbda0142680 auth_method 0 2024-02-06T21:41:42.153 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.149+0000 7fbd9ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:42.153 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.149+0000 7fbd9ed9d640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:42.153 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.149+0000 7fbd9ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:42.153 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.149+0000 7fbd9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:42.153 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.149+0000 7fbd9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:42.153 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.149+0000 7fbd9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:42.153 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.149+0000 7fbd9ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:42.153 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.149+0000 7fbd9ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:42.153 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.149+0000 7fbd9ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:42.154 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.151+0000 7fbd9e59c640 10 monclient(hunting): handle_auth_done global_id 4372 payload 274 2024-02-06T21:41:42.154 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.151+0000 7fbd9e59c640 1 -- 172.21.15.120:0/1366153902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbda013f830 msgr2=0x7fbda013fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:42.154 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.151+0000 7fbd9e59c640 1 --2- 172.21.15.120:0/1366153902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbda013f830 0x7fbda013fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.154 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.151+0000 7fbd9e59c640 1 -- 172.21.15.120:0/1366153902 >> v2:172.21.15.120:3300/0 conn(0x7fbda0142680 msgr2=0x7fbda0142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:42.154 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.151+0000 7fbd9e59c640 1 --2- 172.21.15.120:0/1366153902 >> v2:172.21.15.120:3300/0 conn(0x7fbda0142680 0x7fbda0142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.154 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.151+0000 7fbd9e59c640 10 monclient: _finish_hunting 0 2024-02-06T21:41:42.154 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.151+0000 7fbd9e59c640 1 monclient: found mon.noname-c 2024-02-06T21:41:42.155 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.151+0000 7fbd9e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:42.155 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.151+0000 7fbd9e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T21:41:42.155 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.151+0000 7fbd9e59c640 1 -- 172.21.15.120:0/1366153902 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda00ab720 con 0x7fbda00b54b0 2024-02-06T21:41:42.155 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.152+0000 7fbd9e59c640 1 --2- 172.21.15.120:0/1366153902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda00b54b0 0x7fbda00a9860 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbda006c7b0 tx=0x7fbd9002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0b5930c9ebf3d46 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbd9d59a640 1 -- 172.21.15.120:0/1366153902 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90031640 con 0x7fbda00b54b0 2024-02-06T21:41:42.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbd9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbd9d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:42.158 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbd9d59a640 10 monclient: dump: 2024-02-06T21:41:42.158 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.158 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.158 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.158 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.159 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.159 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.159 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.159 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.159 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.159 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbd9d59a640 10 monclient: _finish_auth 0 2024-02-06T21:41:42.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbd9d59a640 10 monclient: _check_auth_tickets 2024-02-06T21:41:42.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbd9d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:42.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbd9d59a640 1 -- 172.21.15.120:0/1366153902 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd900317a0 con 0x7fbda00b54b0 2024-02-06T21:41:42.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbd9d59a640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:42.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbd9d59a640 1 -- 172.21.15.120:0/1366153902 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90031b30 con 0x7fbda00b54b0 2024-02-06T21:41:42.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbda4806640 5 monclient: authenticate success, global_id 4372 2024-02-06T21:41:42.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbda4806640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:42.160 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbda4806640 4 set_mon_vals no callback set 2024-02-06T21:41:42.160 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbd9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.160 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbd9d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:42.160 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.156+0000 7fbd9d59a640 10 monclient: dump: 2024-02-06T21:41:42.160 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.160 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.160 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.160 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.160 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.160 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.160 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.160 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.161 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.161 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.163 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.157+0000 7fbda4806640 10 monclient: shutdown 2024-02-06T21:41:42.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.157+0000 7fbda4806640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:42.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.157+0000 7fbda4806640 1 -- 172.21.15.120:0/1366153902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda00b54b0 msgr2=0x7fbda00a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.157+0000 7fbda4806640 1 --2- 172.21.15.120:0/1366153902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda00b54b0 0x7fbda00a9860 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbda006c7b0 tx=0x7fbd9002b8f0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.157+0000 7fbda4806640 1 -- 172.21.15.120:0/1366153902 shutdown_connections 2024-02-06T21:41:42.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.157+0000 7fbda4806640 1 --2- 172.21.15.120:0/1366153902 >> v2:172.21.15.120:3300/0 conn(0x7fbda0142680 0x7fbda0142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.157+0000 7fbda4806640 1 --2- 172.21.15.120:0/1366153902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbda013f830 0x7fbda013fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.157+0000 7fbda4806640 1 --2- 172.21.15.120:0/1366153902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda00b54b0 0x7fbda00a9860 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.157+0000 7fbda4806640 1 -- 172.21.15.120:0/1366153902 >> 172.21.15.120:0/1366153902 conn(0x7fbda00b8660 msgr2=0x7fbda00b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:42.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.157+0000 7fbda4806640 1 -- 172.21.15.120:0/1366153902 shutdown_connections 2024-02-06T21:41:42.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.157+0000 7fbda4806640 1 -- 172.21.15.120:0/1366153902 wait complete. 2024-02-06T21:41:42.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.158+0000 7fbda4806640 10 monclient: build_initial_monmap 2024-02-06T21:41:42.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.158+0000 7fbda4806640 10 monclient: monmap: 2024-02-06T21:41:42.164 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:42.165 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:42.165 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:42.165 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:42.165 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:42.165 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.165 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:42.165 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:42.165 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:42.165 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.158+0000 7fbda4806640 1 librados: starting msgr at 2024-02-06T21:41:42.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.158+0000 7fbda4806640 1 librados: starting objecter 2024-02-06T21:41:42.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.158+0000 7fbda4806640 1 Processor -- start 2024-02-06T21:41:42.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.158+0000 7fbda4806640 1 -- start start 2024-02-06T21:41:42.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.158+0000 7fbda4806640 1 librados: setting wanted keys 2024-02-06T21:41:42.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.158+0000 7fbda4806640 1 librados: calling monclient init 2024-02-06T21:41:42.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.158+0000 7fbda4806640 10 monclient: init 2024-02-06T21:41:42.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:42.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T21:41:42.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda00b54b0 0x7fbda00b7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 10 monclient(hunting): picked mon.noname-b con 0x7fbda00b54b0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:42.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbda013f830 0x7fbda00b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 10 monclient(hunting): picked mon.noname-c con 0x7fbda013f830 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:42.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbda0142680 0x7fbda01d7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 10 monclient(hunting): picked mon.noname-a con 0x7fbda0142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:42.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00b0d20 con 0x7fbda0142680 2024-02-06T21:41:42.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00b0350 con 0x7fbda00b54b0 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00af570 con 0x7fbda013f830 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbda4806640 10 monclient(hunting): authenticate will time out at 1332.496717s 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbd9e59c640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda00b54b0 0x7fbda00b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.159+0000 7fbd9ed9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbda0142680 0x7fbda01d7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9e59c640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda00b54b0 0x7fbda00b7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:38924/0 (socket says 172.21.15.120:38924) 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9e59c640 1 -- 172.21.15.120:0/1096688207 learned_addr learned my addr 172.21.15.120:0/1096688207 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9e59c640 10 monclient(hunting): get_auth_request con 0x7fbda00b54b0 auth_method 0 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9e59c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9ed9d640 10 monclient(hunting): get_auth_request con 0x7fbda0142680 auth_method 0 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9ed9d640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:42.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:42.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:42.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:42.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:42.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:42.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:42.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.160+0000 7fbd9ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:42.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.163+0000 7fbd9e59c640 10 monclient(hunting): handle_auth_done global_id 4375 payload 824 2024-02-06T21:41:42.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.163+0000 7fbd9e59c640 1 -- 172.21.15.120:0/1096688207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbda013f830 msgr2=0x7fbda00b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.163+0000 7fbd9e59c640 1 --2- 172.21.15.120:0/1096688207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbda013f830 0x7fbda00b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.163+0000 7fbd9e59c640 1 -- 172.21.15.120:0/1096688207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbda0142680 msgr2=0x7fbda01d7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.163+0000 7fbd9e59c640 1 --2- 172.21.15.120:0/1096688207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbda0142680 0x7fbda01d7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.163+0000 7fbd9e59c640 10 monclient: _finish_hunting 0 2024-02-06T21:41:42.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.163+0000 7fbd9e59c640 1 monclient: found mon.noname-b 2024-02-06T21:41:42.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.163+0000 7fbd9e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:42.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.163+0000 7fbd9e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:42.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.163+0000 7fbd9e59c640 1 -- 172.21.15.120:0/1096688207 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd900094d0 con 0x7fbda00b54b0 2024-02-06T21:41:42.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.165+0000 7fbd9e59c640 1 --2- 172.21.15.120:0/1096688207 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda00b54b0 0x7fbda00b7140 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbd90000c00 tx=0x7fbd900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.165+0000 7fbd8effd640 10 client.?.objecter ms_handle_connect 0x7fbda00b54b0 2024-02-06T21:41:42.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.165+0000 7fbd8effd640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:42.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbd8effd640 1 -- 172.21.15.120:0/1096688207 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9003a930 con 0x7fbda00b54b0 2024-02-06T21:41:42.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbd8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbd8effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:42.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbd8effd640 10 monclient: dump: 2024-02-06T21:41:42.171 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.172 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.172 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.172 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.172 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.172 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.172 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.172 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.172 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.172 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbd8effd640 10 monclient: _finish_auth 0 2024-02-06T21:41:42.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbd8effd640 10 monclient: _check_auth_tickets 2024-02-06T21:41:42.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbd8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:42.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbda4806640 5 monclient: authenticate success, global_id 4375 2024-02-06T21:41:42.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbda4806640 10 monclient: _renew_subs 2024-02-06T21:41:42.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbda4806640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:42.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbda4806640 1 -- 172.21.15.120:0/1096688207 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda00af7e0 con 0x7fbda00b54b0 2024-02-06T21:41:42.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbda4806640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:42.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbda4806640 10 monclient: _renew_subs 2024-02-06T21:41:42.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbda4806640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:42.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbda4806640 1 -- 172.21.15.120:0/1096688207 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda00afee0 con 0x7fbda00b54b0 2024-02-06T21:41:42.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.167+0000 7fbda4806640 1 librados: init done 2024-02-06T21:41:42.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.168+0000 7fbda4806640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.168+0000 7fbda4806640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.168+0000 7fbda4806640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:42.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.168+0000 7fbda4806640 1 -- 172.21.15.120:0/1096688207 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda004ef90 con 0x7fbda00b54b0 2024-02-06T21:41:42.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.211+0000 7f86abf0e640 10 monclient: get_monmap_and_config 2024-02-06T21:41:42.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.211+0000 7f86abf0e640 10 monclient: build_initial_monmap 2024-02-06T21:41:42.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.211+0000 7f86abf0e640 10 monclient: monmap: 2024-02-06T21:41:42.213 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:42.213 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.213 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:42.212761+0000 2024-02-06T21:41:42.213 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:42.212761+0000 2024-02-06T21:41:42.213 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:42.214 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.214 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:42.214 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:42.214 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:42.214 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:42.214 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:42.214 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 1 Processor -- start 2024-02-06T21:41:42.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 1 -- start start 2024-02-06T21:41:42.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 10 monclient: init 2024-02-06T21:41:42.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:42.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-02-06T21:41:42.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f86a40ae8e0 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:42.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f86a40aecc0 0x7f86a406f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 10 monclient(hunting): picked mon.noname-a con 0x7f86a40aecc0 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:42.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f86a4072540 0x7f86a40729b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 10 monclient(hunting): picked mon.noname-c con 0x7f86a4072540 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:42.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a40718d0 con 0x7f86a40ae8e0 2024-02-06T21:41:42.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a4071b20 con 0x7f86a40aecc0 2024-02-06T21:41:42.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a406ff30 con 0x7f86a4072540 2024-02-06T21:41:42.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:42.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.212+0000 7f86abf0e640 10 monclient(hunting): authenticate will time out at 1332.550373s 2024-02-06T21:41:42.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.213+0000 7f86a94a9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f86a40aecc0 0x7f86a406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.213+0000 7f86a94a9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f86a40aecc0 0x7f86a406f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60436/0 (socket says 172.21.15.120:60436) 2024-02-06T21:41:42.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.213+0000 7f86a94a9640 1 -- 172.21.15.120:0/630209444 learned_addr learned my addr 172.21.15.120:0/630209444 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:42.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.213+0000 7f86a94a9640 10 monclient(hunting): get_auth_request con 0x7f86a40aecc0 auth_method 0 2024-02-06T21:41:42.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.213+0000 7f86a94a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:42.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.213+0000 7f86a94a9640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:42.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.213+0000 7f86a94a9640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:42.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.214+0000 7f86a94a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:42.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.214+0000 7f86a94a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:42.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.214+0000 7f86a94a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:42.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.216+0000 7f86a94a9640 10 monclient(hunting): handle_auth_done global_id 4371 payload 274 2024-02-06T21:41:42.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.216+0000 7f86a94a9640 1 -- 172.21.15.120:0/630209444 >> v2:172.21.15.135:3300/0 conn(0x7f86a4072540 msgr2=0x7f86a40729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:42.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.216+0000 7f86a94a9640 1 --2- 172.21.15.120:0/630209444 >> v2:172.21.15.135:3300/0 conn(0x7f86a4072540 0x7f86a40729b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.216+0000 7f86a94a9640 1 -- 172.21.15.120:0/630209444 >> v1:172.21.15.135:6789/0 conn(0x7f86a40ae8e0 legacy=0x7f86a41431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:42.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.216+0000 7f86a94a9640 10 monclient: _finish_hunting 0 2024-02-06T21:41:42.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.216+0000 7f86a94a9640 1 monclient: found mon.noname-a 2024-02-06T21:41:42.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.216+0000 7f86a94a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:42.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.216+0000 7f86a94a9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:42.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.216+0000 7f86a94a9640 1 -- 172.21.15.120:0/630209444 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a406fba0 con 0x7f86a40aecc0 2024-02-06T21:41:42.220 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.218+0000 7f86a94a9640 1 --2- 172.21.15.120:0/630209444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a40aecc0 0x7f86a406f660 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f86a406f540 tx=0x7f869402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a180ee6ca94923b7 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.223 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86a8ca8640 1 -- 172.21.15.120:0/630209444 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86940317e0 con 0x7f86a40aecc0 2024-02-06T21:41:42.224 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86a8ca8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.224 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86a8ca8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:42.224 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86a8ca8640 10 monclient: dump: 2024-02-06T21:41:42.224 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.224 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.224 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.224 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.224 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.224 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.224 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.224 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.224 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.224 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86a8ca8640 10 monclient: _finish_auth 0 2024-02-06T21:41:42.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86a8ca8640 10 monclient: _check_auth_tickets 2024-02-06T21:41:42.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86a8ca8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:42.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86a8ca8640 1 -- 172.21.15.120:0/630209444 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8694031940 con 0x7f86a40aecc0 2024-02-06T21:41:42.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86abf0e640 5 monclient: authenticate success, global_id 4371 2024-02-06T21:41:42.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86abf0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:42.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86a8ca8640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:42.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86a8ca8640 1 -- 172.21.15.120:0/630209444 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8694031cd0 con 0x7f86a40aecc0 2024-02-06T21:41:42.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86abf0e640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:42.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86a8ca8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86abf0e640 4 set_mon_vals no callback set 2024-02-06T21:41:42.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86a8ca8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:42.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86a8ca8640 10 monclient: dump: 2024-02-06T21:41:42.225 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.225 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.226 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.226 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.226 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.226 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.226 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.226 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.226 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.226 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.222+0000 7f86abf0e640 10 monclient: shutdown 2024-02-06T21:41:42.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.223+0000 7f86abf0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:42.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.223+0000 7f86abf0e640 1 -- 172.21.15.120:0/630209444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a40aecc0 msgr2=0x7f86a406f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.223+0000 7f86abf0e640 1 --2- 172.21.15.120:0/630209444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a40aecc0 0x7f86a406f660 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f86a406f540 tx=0x7f869402b880 comp rx=0 tx=0).stop 2024-02-06T21:41:42.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.223+0000 7f86abf0e640 1 -- 172.21.15.120:0/630209444 shutdown_connections 2024-02-06T21:41:42.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.223+0000 7f86abf0e640 1 --2- 172.21.15.120:0/630209444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a40aecc0 0x7f86a406f660 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.223+0000 7f86abf0e640 1 --2- 172.21.15.120:0/630209444 >> v2:172.21.15.135:3300/0 conn(0x7f86a4072540 0x7f86a40729b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.223+0000 7f86abf0e640 1 -- 172.21.15.120:0/630209444 >> 172.21.15.120:0/630209444 conn(0x7f86a40b0bc0 msgr2=0x7f86a40b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:42.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.223+0000 7f86abf0e640 1 -- 172.21.15.120:0/630209444 shutdown_connections 2024-02-06T21:41:42.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.223+0000 7f86abf0e640 1 -- 172.21.15.120:0/630209444 wait complete. 2024-02-06T21:41:42.229 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.223+0000 7f86abf0e640 10 monclient: build_initial_monmap 2024-02-06T21:41:42.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.223+0000 7f86abf0e640 10 monclient: monmap: 2024-02-06T21:41:42.230 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:42.230 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:42.230 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:42.230 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:42.230 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:42.230 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.230 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:42.230 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:42.230 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:42.230 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.224+0000 7f86abf0e640 1 librados: starting msgr at 2024-02-06T21:41:42.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.224+0000 7f86abf0e640 1 librados: starting objecter 2024-02-06T21:41:42.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.224+0000 7f86abf0e640 1 Processor -- start 2024-02-06T21:41:42.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.224+0000 7f86abf0e640 1 -- start start 2024-02-06T21:41:42.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.224+0000 7f86abf0e640 1 librados: setting wanted keys 2024-02-06T21:41:42.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.224+0000 7f86abf0e640 1 librados: calling monclient init 2024-02-06T21:41:42.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.224+0000 7f86abf0e640 10 monclient: init 2024-02-06T21:41:42.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:42.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T21:41:42.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f86a4072540 0x7f86a40a9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 10 monclient(hunting): picked mon.noname-b con 0x7f86a4072540 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:42.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a40ae8e0 0x7f86a40ab850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 10 monclient(hunting): picked mon.noname-c con 0x7f86a40ae8e0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:42.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a40aa1f0 0x7f86a40abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 10 monclient(hunting): picked mon.noname-a con 0x7f86a40aa1f0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:42.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a41367f0 con 0x7f86a40aa1f0 2024-02-06T21:41:42.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.232 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a4136c80 con 0x7f86a4072540 2024-02-06T21:41:42.232 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.232 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a40ab1c0 con 0x7f86a40ae8e0 2024-02-06T21:41:42.232 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:42.232 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86abf0e640 10 monclient(hunting): authenticate will time out at 1332.562591s 2024-02-06T21:41:42.232 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.225+0000 7f86aa4ab640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a40aa1f0 0x7f86a40abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.232 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.226+0000 7f86aa4ab640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a40aa1f0 0x7f86a40abd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60444/0 (socket says 172.21.15.120:60444) 2024-02-06T21:41:42.232 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.226+0000 7f86aa4ab640 1 -- 172.21.15.120:0/967099549 learned_addr learned my addr 172.21.15.120:0/967099549 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:42.232 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.226+0000 7f86aa4ab640 10 monclient(hunting): get_auth_request con 0x7f86a40aa1f0 auth_method 0 2024-02-06T21:41:42.232 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.226+0000 7f86aa4ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:42.232 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.226+0000 7f86aa4ab640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:42.232 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.226+0000 7f86aa4ab640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:42.232 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.226+0000 7f86aa4ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:42.233 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.226+0000 7f86aa4ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:42.233 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.226+0000 7f86aa4ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:42.233 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.229+0000 7f86aa4ab640 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2024-02-06T21:41:42.234 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.229+0000 7f86aa4ab640 1 -- 172.21.15.120:0/967099549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a40ae8e0 msgr2=0x7f86a40ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.234 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.229+0000 7f86aa4ab640 1 --2- 172.21.15.120:0/967099549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a40ae8e0 0x7f86a40ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.234 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.229+0000 7f86aa4ab640 1 -- 172.21.15.120:0/967099549 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f86a4072540 msgr2=0x7f86a40a9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.234 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.229+0000 7f86aa4ab640 1 --2- 172.21.15.120:0/967099549 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f86a4072540 0x7f86a40a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.234 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.229+0000 7f86aa4ab640 10 monclient: _finish_hunting 0 2024-02-06T21:41:42.234 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.229+0000 7f86aa4ab640 1 monclient: found mon.noname-a 2024-02-06T21:41:42.234 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.229+0000 7f86aa4ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:42.234 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.229+0000 7f86aa4ab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:42.234 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.229+0000 7f86aa4ab640 1 -- 172.21.15.120:0/967099549 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86940094d0 con 0x7f86a40aa1f0 2024-02-06T21:41:42.234 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.231+0000 7f86aa4ab640 1 --2- 172.21.15.120:0/967099549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a40aa1f0 0x7f86a40abd90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f869c00b9a0 tx=0x7f869c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.234 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.231+0000 7f869a7fc640 10 client.?.objecter ms_handle_connect 0x7f86a40aa1f0 2024-02-06T21:41:42.234 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.231+0000 7f869a7fc640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:42.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.231+0000 7f8003fff640 1 -- 172.21.15.120:0/2504394258 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8014058650 con 0x7f801c142bb0 2024-02-06T21:41:42.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.231+0000 7f8003fff640 10 client.4328.objecter ms_dispatch 0x7f801c000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:42.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.231+0000 7f8003fff640 3 client.4328.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:42.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.231+0000 7f8003fff640 3 client.4328.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:42.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.231+0000 7f8003fff640 20 client.4328.objecter dump_active .. 0 homeless 2024-02-06T21:41:42.236 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.232+0000 7f8003fff640 1 -- 172.21.15.120:0/2504394258 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f80140548f0 con 0x7f801c142bb0 2024-02-06T21:41:42.236 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.232+0000 7f8003fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.236 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.232+0000 7f8003fff640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:42.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.235+0000 7f869a7fc640 1 -- 172.21.15.120:0/967099549 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c00cba0 con 0x7f86a40aa1f0 2024-02-06T21:41:42.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f869a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f869a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:42.238 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f869a7fc640 10 monclient: dump: 2024-02-06T21:41:42.238 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.238 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.238 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.238 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.238 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.238 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.238 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.238 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.239 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.239 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f869a7fc640 10 monclient: _finish_auth 0 2024-02-06T21:41:42.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f869a7fc640 10 monclient: _check_auth_tickets 2024-02-06T21:41:42.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f869a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:42.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f869a7fc640 1 -- 172.21.15.120:0/967099549 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c00cd20 con 0x7f86a40aa1f0 2024-02-06T21:41:42.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f86abf0e640 5 monclient: authenticate success, global_id 4374 2024-02-06T21:41:42.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f86abf0e640 10 monclient: _renew_subs 2024-02-06T21:41:42.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f86abf0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:42.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f86abf0e640 1 -- 172.21.15.120:0/967099549 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a40ab490 con 0x7f86a40aa1f0 2024-02-06T21:41:42.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f869a7fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:42.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f869bfff640 4 set_mon_vals no callback set 2024-02-06T21:41:42.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f86abf0e640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:42.239 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f86abf0e640 10 monclient: _renew_subs 2024-02-06T21:41:42.240 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f86abf0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:42.240 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f869a7fc640 1 -- 172.21.15.120:0/967099549 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c014680 con 0x7f86a40aa1f0 2024-02-06T21:41:42.240 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f86abf0e640 1 -- 172.21.15.120:0/967099549 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a40a9bd0 con 0x7f86a40aa1f0 2024-02-06T21:41:42.240 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f86abf0e640 1 librados: init done 2024-02-06T21:41:42.240 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f869a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.240 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f869a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:42.240 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.236+0000 7f869a7fc640 10 monclient: dump: 2024-02-06T21:41:42.240 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.240 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.240 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.240 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.240 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.240 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.240 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.241 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.241 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.241 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.241 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.237+0000 7f86abf0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.241 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.237+0000 7f86abf0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.241 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.237+0000 7f86abf0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:42.241 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.237+0000 7f86abf0e640 1 -- 172.21.15.120:0/967099549 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a41369a0 con 0x7f86a40aa1f0 2024-02-06T21:41:42.260 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.257+0000 7f869a7fc640 1 -- 172.21.15.120:0/967099549 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f869c01d3b0 con 0x7f86a40aa1f0 2024-02-06T21:41:42.260 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.257+0000 7f869a7fc640 1 --2- 172.21.15.120:0/967099549 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f868806d220 0x7f868806f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.260 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.258+0000 7f86a9caa640 1 --2- 172.21.15.120:0/967099549 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f868806d220 0x7f868806f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.261 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.258+0000 7f86a9caa640 10 monclient: get_auth_request con 0x7f868806d220 auth_method 0 2024-02-06T21:41:42.261 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.258+0000 7f869a7fc640 10 client.4374.objecter ms_handle_connect 0x7f868806d220 2024-02-06T21:41:42.261 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.258+0000 7f86a9caa640 1 --2- 172.21.15.120:0/967099549 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f868806d220 0x7f868806f6e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f86a0001560 tx=0x7f86a0007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.262 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.260+0000 7f869a7fc640 1 -- 172.21.15.120:0/967099549 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f869c0592a0 con 0x7f86a40aa1f0 2024-02-06T21:41:42.262 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.260+0000 7f869a7fc640 10 client.4374.objecter ms_dispatch 0x7f86a4000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:42.262 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.260+0000 7f869a7fc640 3 client.4374.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:42.262 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.260+0000 7f869a7fc640 3 client.4374.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:42.262 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.260+0000 7f869a7fc640 20 client.4374.objecter dump_active .. 0 homeless 2024-02-06T21:41:42.285 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-02-06T21:41:42.383 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-02-06T21:41:42.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.388+0000 7f752f63e640 10 monclient: get_monmap_and_config 2024-02-06T21:41:42.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.388+0000 7f752f63e640 10 monclient: build_initial_monmap 2024-02-06T21:41:42.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.388+0000 7f752f63e640 10 monclient: monmap: 2024-02-06T21:41:42.390 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:42.390 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.390 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:42.389855+0000 2024-02-06T21:41:42.390 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:42.389855+0000 2024-02-06T21:41:42.390 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:42.390 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.390 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:42.390 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:42.390 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:42.391 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:42.391 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:42.391 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.389+0000 7f752f63e640 1 Processor -- start 2024-02-06T21:41:42.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.389+0000 7f752f63e640 1 -- start start 2024-02-06T21:41:42.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.389+0000 7f752f63e640 10 monclient: init 2024-02-06T21:41:42.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.389+0000 7f752f63e640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:42.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.389+0000 7f752f63e640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-02-06T21:41:42.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.389+0000 7f752f63e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75280b54b0 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:42.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.389+0000 7f752f63e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f752813ff40 0x7f75280a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.390+0000 7f752f63e640 10 monclient(hunting): picked mon.noname-b con 0x7f752813ff40 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:42.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.390+0000 7f752f63e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7528142680 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:42.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.390+0000 7f752f63e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7528143200 con 0x7f7528142680 2024-02-06T21:41:42.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.390+0000 7f752f63e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75280ab720 con 0x7f75280b54b0 2024-02-06T21:41:42.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.390+0000 7f752f63e640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.390+0000 7f752f63e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75280ab520 con 0x7f752813ff40 2024-02-06T21:41:42.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.390+0000 7f752f63e640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:42.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.390+0000 7f752f63e640 10 monclient(hunting): authenticate will time out at 1332.727644s 2024-02-06T21:41:42.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.390+0000 7f752cbd9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f752813ff40 0x7f75280a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.391+0000 7f752cbd9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f752813ff40 0x7f75280a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57198/0 (socket says 172.21.15.120:57198) 2024-02-06T21:41:42.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.391+0000 7f752cbd9640 1 -- 172.21.15.120:0/4179409083 learned_addr learned my addr 172.21.15.120:0/4179409083 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:42.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.391+0000 7f752cbd9640 10 monclient(hunting): get_auth_request con 0x7f752813ff40 auth_method 0 2024-02-06T21:41:42.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.391+0000 7f752cbd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:42.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.391+0000 7f752cbd9640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:42.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.391+0000 7f752cbd9640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:42.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.391+0000 7f752cbd9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:42.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.391+0000 7f752cbd9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:42.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.392+0000 7f752cbd9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:42.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.394+0000 7f752cbd9640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2024-02-06T21:41:42.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.394+0000 7f752cbd9640 1 -- 172.21.15.120:0/4179409083 >> v1:172.21.15.135:6789/0 conn(0x7f75280b54b0 legacy=0x7f752813f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:42.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.394+0000 7f752cbd9640 1 -- 172.21.15.120:0/4179409083 >> v1:172.21.15.120:6789/0 conn(0x7f7528142680 legacy=0x7f7528142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:42.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.394+0000 7f752cbd9640 10 monclient: _finish_hunting 0 2024-02-06T21:41:42.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.394+0000 7f752cbd9640 1 monclient: found mon.noname-b 2024-02-06T21:41:42.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.394+0000 7f752cbd9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:42.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.394+0000 7f752cbd9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:42.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.394+0000 7f752cbd9640 1 -- 172.21.15.120:0/4179409083 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75280b4800 con 0x7f752813ff40 2024-02-06T21:41:42.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.397+0000 7f752cbd9640 1 --2- 172.21.15.120:0/4179409083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f752813ff40 0x7f75280a9860 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f75180098a0 tx=0x7f751802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b76c1d6e5852d6c7 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f751ffff640 1 -- 172.21.15.120:0/4179409083 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7518031640 con 0x7f752813ff40 2024-02-06T21:41:42.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f751ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f751ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:42.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f751ffff640 10 monclient: dump: 2024-02-06T21:41:42.404 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.404 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.404 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.404 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.404 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.404 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.404 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.404 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.404 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.404 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f751ffff640 10 monclient: _finish_auth 0 2024-02-06T21:41:42.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f751ffff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:42.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f751ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:42.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f751ffff640 1 -- 172.21.15.120:0/4179409083 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75180317a0 con 0x7f752813ff40 2024-02-06T21:41:42.405 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f752f63e640 5 monclient: authenticate success, global_id 4331 2024-02-06T21:41:42.405 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f752f63e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:42.405 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f751ffff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:42.405 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f752f63e640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:42.405 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f752f63e640 4 set_mon_vals no callback set 2024-02-06T21:41:42.405 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f751ffff640 1 -- 172.21.15.120:0/4179409083 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7518031b30 con 0x7f752813ff40 2024-02-06T21:41:42.405 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f751ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.405 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f751ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:42.405 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f751ffff640 10 monclient: dump: 2024-02-06T21:41:42.405 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.405 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.405 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.405 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.405 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.405 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.406 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.406 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.406 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.406 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f752f63e640 10 monclient: shutdown 2024-02-06T21:41:42.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f752f63e640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:42.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.402+0000 7f752f63e640 1 -- 172.21.15.120:0/4179409083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f752813ff40 msgr2=0x7f75280a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.403+0000 7f752f63e640 1 --2- 172.21.15.120:0/4179409083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f752813ff40 0x7f75280a9860 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f75180098a0 tx=0x7f751802b8f0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.403+0000 7f752f63e640 1 -- 172.21.15.120:0/4179409083 shutdown_connections 2024-02-06T21:41:42.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.403+0000 7f752f63e640 1 --2- 172.21.15.120:0/4179409083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f752813ff40 0x7f75280a9860 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.403+0000 7f752f63e640 1 -- 172.21.15.120:0/4179409083 >> 172.21.15.120:0/4179409083 conn(0x7f75280b8660 msgr2=0x7f75280b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:42.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.403+0000 7f752f63e640 1 -- 172.21.15.120:0/4179409083 shutdown_connections 2024-02-06T21:41:42.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.403+0000 7f752f63e640 1 -- 172.21.15.120:0/4179409083 wait complete. 2024-02-06T21:41:42.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.404+0000 7f752f63e640 10 monclient: build_initial_monmap 2024-02-06T21:41:42.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.404+0000 7f752f63e640 10 monclient: monmap: 2024-02-06T21:41:42.409 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:42.409 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:42.409 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:42.409 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:42.409 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:42.409 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.409 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:42.409 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:42.409 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:42.410 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.404+0000 7f752f63e640 1 librados: starting msgr at 2024-02-06T21:41:42.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.404+0000 7f752f63e640 1 librados: starting objecter 2024-02-06T21:41:42.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.404+0000 7f752f63e640 1 Processor -- start 2024-02-06T21:41:42.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.404+0000 7f752f63e640 1 -- start start 2024-02-06T21:41:42.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.404+0000 7f752f63e640 1 librados: setting wanted keys 2024-02-06T21:41:42.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.404+0000 7f752f63e640 1 librados: calling monclient init 2024-02-06T21:41:42.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.404+0000 7f752f63e640 10 monclient: init 2024-02-06T21:41:42.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:42.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T21:41:42.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75280b54b0 0x7f75280b7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 10 monclient(hunting): picked mon.noname-b con 0x7f75280b54b0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:42.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f752813ff40 0x7f75281353c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 10 monclient(hunting): picked mon.noname-a con 0x7f752813ff40 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:42.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7528142680 0x7f75281375f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 10 monclient(hunting): picked mon.noname-c con 0x7f7528142680 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:42.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7528133ef0 con 0x7f752813ff40 2024-02-06T21:41:42.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75281365e0 con 0x7f75280b54b0 2024-02-06T21:41:42.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7528134400 con 0x7f7528142680 2024-02-06T21:41:42.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:42.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752f63e640 10 monclient(hunting): authenticate will time out at 1332.742778s 2024-02-06T21:41:42.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.405+0000 7f752dbdb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7528142680 0x7f75281375f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.406+0000 7f752dbdb640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7528142680 0x7f75281375f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57212/0 (socket says 172.21.15.120:57212) 2024-02-06T21:41:42.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.406+0000 7f752dbdb640 1 -- 172.21.15.120:0/357746855 learned_addr learned my addr 172.21.15.120:0/357746855 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:42.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.406+0000 7f752dbdb640 10 monclient(hunting): get_auth_request con 0x7f7528142680 auth_method 0 2024-02-06T21:41:42.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.406+0000 7f752dbdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:42.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.406+0000 7f752dbdb640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:42.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.406+0000 7f752dbdb640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:42.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.406+0000 7f752dbdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:42.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.406+0000 7f752dbdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:42.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.406+0000 7f752dbdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:42.413 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.409+0000 7fbd8effd640 1 -- 172.21.15.120:0/1096688207 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9003aa90 con 0x7fbda00b54b0 2024-02-06T21:41:42.413 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.409+0000 7fbd8effd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:42.413 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.409+0000 7fbd8effd640 1 -- 172.21.15.120:0/1096688207 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9003ae00 con 0x7fbda00b54b0 2024-02-06T21:41:42.413 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.409+0000 7fbd8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.413 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.409+0000 7fbd8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:42.413 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.409+0000 7fbd8effd640 10 monclient: dump: 2024-02-06T21:41:42.413 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.413 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.413 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.413 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.413 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.413 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.414 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.414 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.414 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.414 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.409+0000 7fbd9cd99640 4 set_mon_vals no callback set 2024-02-06T21:41:42.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.410+0000 7fbd8effd640 1 -- 172.21.15.120:0/1096688207 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbd9002f050 con 0x7fbda00b54b0 2024-02-06T21:41:42.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.410+0000 7fbd8effd640 1 --2- 172.21.15.120:0/1096688207 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbd8006d1d0 0x7fbd8006f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.411+0000 7fbd9dd9b640 1 --2- 172.21.15.120:0/1096688207 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbd8006d1d0 0x7fbd8006f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.411+0000 7fbd9dd9b640 10 monclient: get_auth_request con 0x7fbd8006d1d0 auth_method 0 2024-02-06T21:41:42.414 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.411+0000 7f752dbdb640 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2024-02-06T21:41:42.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.411+0000 7fbd9dd9b640 1 --2- 172.21.15.120:0/1096688207 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbd8006d1d0 0x7fbd8006f690 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbda013d0c0 tx=0x7fbd94006c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.411+0000 7f752dbdb640 1 -- 172.21.15.120:0/357746855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75280b54b0 msgr2=0x7f75280b7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.411+0000 7f752dbdb640 1 --2- 172.21.15.120:0/357746855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75280b54b0 0x7f75280b7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.411+0000 7f752dbdb640 1 -- 172.21.15.120:0/357746855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f752813ff40 msgr2=0x7f75281353c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.411+0000 7f752dbdb640 1 --2- 172.21.15.120:0/357746855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f752813ff40 0x7f75281353c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.411+0000 7f752dbdb640 10 monclient: _finish_hunting 0 2024-02-06T21:41:42.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.411+0000 7f752dbdb640 1 monclient: found mon.noname-c 2024-02-06T21:41:42.416 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.411+0000 7fbd8effd640 10 client.4375.objecter ms_handle_connect 0x7fbd8006d1d0 2024-02-06T21:41:42.416 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.411+0000 7f752dbdb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:42.416 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.411+0000 7f752dbdb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:41:42.416 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.411+0000 7f752dbdb640 1 -- 172.21.15.120:0/357746855 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75180094d0 con 0x7f7528142680 2024-02-06T21:41:42.416 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.413+0000 7fbd8effd640 1 -- 172.21.15.120:0/1096688207 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbd9007b290 con 0x7fbda00b54b0 2024-02-06T21:41:42.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.413+0000 7fbd8effd640 10 client.4375.objecter ms_dispatch 0x7fbda0000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:42.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.413+0000 7fbd8effd640 3 client.4375.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:42.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.413+0000 7fbd8effd640 3 client.4375.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:42.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.414+0000 7fbd8effd640 20 client.4375.objecter dump_active .. 0 homeless 2024-02-06T21:41:42.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.415+0000 7f752dbdb640 1 --2- 172.21.15.120:0/357746855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7528142680 0x7f75281375f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f752000c7b0 tx=0x7f752000cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.415+0000 7f751dffb640 10 client.?.objecter ms_handle_connect 0x7f7528142680 2024-02-06T21:41:42.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.415+0000 7f751dffb640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:42.426 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.422+0000 7f751dffb640 1 -- 172.21.15.120:0/357746855 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7520003d80 con 0x7f7528142680 2024-02-06T21:41:42.426 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f751dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.426 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f751dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:42.426 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f751dffb640 10 monclient: dump: 2024-02-06T21:41:42.426 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.426 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.426 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.426 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.427 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.427 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.427 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.427 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.427 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.427 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f751dffb640 10 monclient: _finish_auth 0 2024-02-06T21:41:42.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f751dffb640 10 monclient: _check_auth_tickets 2024-02-06T21:41:42.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f751dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:42.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f751dffb640 1 -- 172.21.15.120:0/357746855 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7520005070 con 0x7f7528142680 2024-02-06T21:41:42.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f751dffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:42.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f751dffb640 1 -- 172.21.15.120:0/357746855 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7520005420 con 0x7f7528142680 2024-02-06T21:41:42.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f751dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f751dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:42.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f751dffb640 10 monclient: dump: 2024-02-06T21:41:42.428 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.428 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.428 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.428 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.428 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.428 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.428 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.428 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.428 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.428 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f751f7fe640 4 set_mon_vals no callback set 2024-02-06T21:41:42.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f752f63e640 5 monclient: authenticate success, global_id 4334 2024-02-06T21:41:42.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f752f63e640 10 monclient: _renew_subs 2024-02-06T21:41:42.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f752f63e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:42.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f752f63e640 1 -- 172.21.15.120:0/357746855 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7528136850 con 0x7f7528142680 2024-02-06T21:41:42.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f752f63e640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:42.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f752f63e640 10 monclient: _renew_subs 2024-02-06T21:41:42.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f752f63e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:42.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f752f63e640 1 -- 172.21.15.120:0/357746855 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7528136e10 con 0x7f7528142680 2024-02-06T21:41:42.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7f752f63e640 1 librados: init done 2024-02-06T21:41:42.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7fbd8effd640 1 -- 172.21.15.120:0/1096688207 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbd90076e30 con 0x7fbda00b54b0 2024-02-06T21:41:42.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7fbd8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.423+0000 7fbd8effd640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:42.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.424+0000 7f752f63e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.424+0000 7f752f63e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.424+0000 7f752f63e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:42.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.424+0000 7f752f63e640 1 -- 172.21.15.120:0/357746855 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752814df40 con 0x7f7528142680 2024-02-06T21:41:42.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.431+0000 7f8021f04640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-02-06T21:41:42.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.431+0000 7f8021f04640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-02-06T21:41:42.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.431+0000 7f8021f04640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:42.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.431+0000 7f8021f04640 1 -- 172.21.15.120:0/2504394258 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f801c139e00 con 0x7f801c142bb0 2024-02-06T21:41:42.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.448+0000 7f751dffb640 1 -- 172.21.15.120:0/357746855 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7520005580 con 0x7f7528142680 2024-02-06T21:41:42.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.448+0000 7f751dffb640 1 --2- 172.21.15.120:0/357746855 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f750c06d1a0 0x7f750c06f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.449+0000 7f752d3da640 1 --2- 172.21.15.120:0/357746855 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f750c06d1a0 0x7f750c06f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.449+0000 7f752d3da640 10 monclient: get_auth_request con 0x7f750c06d1a0 auth_method 0 2024-02-06T21:41:42.451 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.449+0000 7f752d3da640 1 --2- 172.21.15.120:0/357746855 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f750c06d1a0 0x7f750c06f660 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7524009650 tx=0x7f7524006ba0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.451 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.449+0000 7f751dffb640 10 client.4334.objecter ms_handle_connect 0x7f750c06d1a0 2024-02-06T21:41:42.454 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.453+0000 7f751dffb640 1 -- 172.21.15.120:0/357746855 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7520058610 con 0x7f7528142680 2024-02-06T21:41:42.454 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.453+0000 7f751dffb640 10 client.4334.objecter ms_dispatch 0x7f7528000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:42.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.453+0000 7f751dffb640 3 client.4334.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:42.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.453+0000 7f751dffb640 3 client.4334.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:42.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.453+0000 7f751dffb640 20 client.4334.objecter dump_active .. 0 homeless 2024-02-06T21:41:42.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.506+0000 7faff2aa1640 10 monclient: get_monmap_and_config 2024-02-06T21:41:42.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.506+0000 7faff2aa1640 10 monclient: build_initial_monmap 2024-02-06T21:41:42.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.506+0000 7faff2aa1640 10 monclient: monmap: 2024-02-06T21:41:42.508 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:42.508 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.508 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:42.507899+0000 2024-02-06T21:41:42.508 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:42.507899+0000 2024-02-06T21:41:42.508 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:42.508 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.508 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:42.508 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:42.509 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:42.509 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:42.509 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:42.509 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.507+0000 7faff2aa1640 1 Processor -- start 2024-02-06T21:41:42.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.507+0000 7faff2aa1640 1 -- start start 2024-02-06T21:41:42.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.507+0000 7faff2aa1640 10 monclient: init 2024-02-06T21:41:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.508+0000 7faff2aa1640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.508+0000 7faff2aa1640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-02-06T21:41:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.508+0000 7faff2aa1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fafec070dc0 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.508+0000 7faff2aa1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafec0ad7b0 0x7fafec0b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.508+0000 7faff2aa1640 10 monclient(hunting): picked mon.noname-b con 0x7fafec0ad7b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.508+0000 7faff2aa1640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fafec0711a0 0x7fafec144380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.508+0000 7faff2aa1640 10 monclient(hunting): picked mon.noname-c con 0x7fafec0711a0 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.508+0000 7faff2aa1640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafec0731b0 con 0x7fafec070dc0 2024-02-06T21:41:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.508+0000 7faff2aa1640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.508+0000 7faff2aa1640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafec072da0 con 0x7fafec0711a0 2024-02-06T21:41:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.508+0000 7faff2aa1640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.508+0000 7faff2aa1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec071bd0 con 0x7fafec0ad7b0 2024-02-06T21:41:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.508+0000 7faff2aa1640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.508+0000 7faff2aa1640 10 monclient(hunting): authenticate will time out at 1332.845719s 2024-02-06T21:41:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.508+0000 7faff103e640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fafec0711a0 0x7fafec144380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.509+0000 7faff103e640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fafec0711a0 0x7fafec144380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:50532/0 (socket says 172.21.15.120:50532) 2024-02-06T21:41:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.509+0000 7faff103e640 1 -- 172.21.15.120:0/1046456638 learned_addr learned my addr 172.21.15.120:0/1046456638 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.509+0000 7faff103e640 10 monclient(hunting): get_auth_request con 0x7fafec0711a0 auth_method 0 2024-02-06T21:41:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.509+0000 7faff103e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.509+0000 7faff103e640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.509+0000 7faff103e640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:42.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.511+0000 7faff103e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:42.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.511+0000 7faff103e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:42.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.511+0000 7faff103e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:42.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.513+0000 7faff103e640 10 monclient(hunting): handle_auth_done global_id 4378 payload 274 2024-02-06T21:41:42.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.513+0000 7faff103e640 1 -- 172.21.15.120:0/1046456638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafec0ad7b0 msgr2=0x7fafec0b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:42.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.513+0000 7faff103e640 1 --2- 172.21.15.120:0/1046456638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafec0ad7b0 0x7fafec0b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.513+0000 7faff103e640 1 -- 172.21.15.120:0/1046456638 >> v1:172.21.15.135:6789/0 conn(0x7fafec070dc0 legacy=0x7fafec143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.513+0000 7faff103e640 10 monclient: _finish_hunting 0 2024-02-06T21:41:42.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.513+0000 7faff103e640 1 monclient: found mon.noname-c 2024-02-06T21:41:42.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.514+0000 7faff103e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:42.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.514+0000 7faff103e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T21:41:42.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.514+0000 7faff103e640 1 -- 172.21.15.120:0/1046456638 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec0afaa0 con 0x7fafec0711a0 2024-02-06T21:41:42.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.517+0000 7faff103e640 1 --2- 172.21.15.120:0/1046456638 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec0711a0 0x7fafec144380 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fafe4009e50 tx=0x7fafe402be70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=949b520eefe5c950 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.518+0000 7f869a7fc640 1 -- 172.21.15.120:0/967099549 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f869c054e40 con 0x7f86a40aa1f0 2024-02-06T21:41:42.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.519+0000 7f869a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.519+0000 7f869a7fc640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.519+0000 7fafeb7fe640 1 -- 172.21.15.120:0/1046456638 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe402ca10 con 0x7fafec0711a0 2024-02-06T21:41:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.520+0000 7fafeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.520+0000 7fafeb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.520+0000 7fafeb7fe640 10 monclient: dump: 2024-02-06T21:41:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.522 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.520+0000 7fafeb7fe640 10 monclient: _finish_auth 0 2024-02-06T21:41:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.520+0000 7fafeb7fe640 10 monclient: _check_auth_tickets 2024-02-06T21:41:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.520+0000 7fafeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.520+0000 7faff2aa1640 5 monclient: authenticate success, global_id 4378 2024-02-06T21:41:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.520+0000 7faff2aa1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:42.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.521+0000 7fafeb7fe640 1 -- 172.21.15.120:0/1046456638 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe402cb70 con 0x7fafec0711a0 2024-02-06T21:41:42.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.521+0000 7fafeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:42.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.521+0000 7fafeb7fe640 1 -- 172.21.15.120:0/1046456638 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe40343f0 con 0x7fafec0711a0 2024-02-06T21:41:42.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.521+0000 7fafeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.521+0000 7fafeb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:42.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.521+0000 7fafeb7fe640 10 monclient: dump: 2024-02-06T21:41:42.523 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.523 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.523 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.524 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.524 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.524 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.524 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.524 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.524 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.524 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.521+0000 7faff2aa1640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:42.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.521+0000 7faff2aa1640 4 set_mon_vals no callback set 2024-02-06T21:41:42.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.522+0000 7faff2aa1640 10 monclient: shutdown 2024-02-06T21:41:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.522+0000 7faff2aa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.522+0000 7faff2aa1640 1 -- 172.21.15.120:0/1046456638 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec0711a0 msgr2=0x7fafec144380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.522+0000 7faff2aa1640 1 --2- 172.21.15.120:0/1046456638 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec0711a0 0x7fafec144380 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fafe4009e50 tx=0x7fafe402be70 comp rx=0 tx=0).stop 2024-02-06T21:41:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.522+0000 7faff2aa1640 1 -- 172.21.15.120:0/1046456638 shutdown_connections 2024-02-06T21:41:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.522+0000 7faff2aa1640 1 --2- 172.21.15.120:0/1046456638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafec0ad7b0 0x7fafec0b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.522+0000 7faff2aa1640 1 --2- 172.21.15.120:0/1046456638 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec0711a0 0x7fafec144380 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.522+0000 7faff2aa1640 1 -- 172.21.15.120:0/1046456638 >> 172.21.15.120:0/1046456638 conn(0x7fafec0b6250 msgr2=0x7fafec0b6660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.522+0000 7faff2aa1640 1 -- 172.21.15.120:0/1046456638 shutdown_connections 2024-02-06T21:41:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.522+0000 7faff2aa1640 1 -- 172.21.15.120:0/1046456638 wait complete. 2024-02-06T21:41:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.523+0000 7faff2aa1640 10 monclient: build_initial_monmap 2024-02-06T21:41:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.523+0000 7faff2aa1640 10 monclient: monmap: 2024-02-06T21:41:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:42.527 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.524+0000 7faff2aa1640 1 librados: starting msgr at 2024-02-06T21:41:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.524+0000 7faff2aa1640 1 librados: starting objecter 2024-02-06T21:41:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.524+0000 7faff2aa1640 1 Processor -- start 2024-02-06T21:41:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.524+0000 7faff2aa1640 1 -- start start 2024-02-06T21:41:42.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.524+0000 7faff2aa1640 1 librados: setting wanted keys 2024-02-06T21:41:42.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.524+0000 7faff2aa1640 1 librados: calling monclient init 2024-02-06T21:41:42.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.524+0000 7faff2aa1640 10 monclient: init 2024-02-06T21:41:42.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:42.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T21:41:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec070dc0 0x7fafec0ac140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 10 monclient(hunting): picked mon.noname-b con 0x7fafec070dc0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafec0711a0 0x7fafec1d58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 10 monclient(hunting): picked mon.noname-c con 0x7fafec0711a0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafec0ad7b0 0x7fafec13ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 10 monclient(hunting): picked mon.noname-a con 0x7fafec0ad7b0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec1d6110 con 0x7fafec0ad7b0 2024-02-06T21:41:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec1d5e30 con 0x7fafec070dc0 2024-02-06T21:41:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec13fa40 con 0x7fafec0711a0 2024-02-06T21:41:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.525+0000 7faff2aa1640 10 monclient(hunting): authenticate will time out at 1332.863404s 2024-02-06T21:41:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.526+0000 7faff083d640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec070dc0 0x7fafec0ac140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.526+0000 7faff083d640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec070dc0 0x7fafec0ac140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:50544/0 (socket says 172.21.15.120:50544) 2024-02-06T21:41:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.526+0000 7faff083d640 1 -- 172.21.15.120:0/1209179394 learned_addr learned my addr 172.21.15.120:0/1209179394 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.526+0000 7faff083d640 10 monclient(hunting): get_auth_request con 0x7fafec070dc0 auth_method 0 2024-02-06T21:41:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.526+0000 7faff083d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.526+0000 7faff083d640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.526+0000 7faff083d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.527+0000 7faff083d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.527+0000 7faff083d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.527+0000 7faff083d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:42.531 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-02-06T21:41:42.532 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.529+0000 7faff083d640 10 monclient(hunting): handle_auth_done global_id 4381 payload 824 2024-02-06T21:41:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.529+0000 7faff083d640 1 -- 172.21.15.120:0/1209179394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafec0711a0 msgr2=0x7fafec1d58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.529+0000 7faff083d640 1 --2- 172.21.15.120:0/1209179394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafec0711a0 0x7fafec1d58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.529+0000 7faff083d640 1 -- 172.21.15.120:0/1209179394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafec0ad7b0 msgr2=0x7fafec13ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.529+0000 7faff083d640 1 --2- 172.21.15.120:0/1209179394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafec0ad7b0 0x7fafec13ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.529+0000 7faff083d640 10 monclient: _finish_hunting 0 2024-02-06T21:41:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.529+0000 7faff083d640 1 monclient: found mon.noname-b 2024-02-06T21:41:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.529+0000 7faff083d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.529+0000 7faff083d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.529+0000 7faff083d640 1 -- 172.21.15.120:0/1209179394 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe4009b30 con 0x7fafec070dc0 2024-02-06T21:41:42.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.531+0000 7faff083d640 1 --2- 172.21.15.120:0/1209179394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec070dc0 0x7fafec0ac140 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fafe000c400 tx=0x7fafe000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.531+0000 7fafe97fa640 10 client.?.objecter ms_handle_connect 0x7fafec070dc0 2024-02-06T21:41:42.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.532+0000 7fafe97fa640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:42.535 INFO:teuthology.orchestra.run.smithi120.stdout:73014444041 2024-02-06T21:41:42.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.533+0000 7f8003fff640 1 -- 172.21.15.120:0/2504394258 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8014017070 con 0x7f801c142bb0 2024-02-06T21:41:42.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.533+0000 7f8003fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-02-06T21:41:42.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.533+0000 7f8003fff640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:42.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7f8021f04640 10 librados: watch_flush enter 2024-02-06T21:41:42.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7f8021f04640 10 librados: watch_flush exit 2024-02-06T21:41:42.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7f8021f04640 20 client.4328.objecter shutdown clearing up homeless session... 2024-02-06T21:41:42.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7f8021f04640 10 client.4328.objecter successfully canceled tick 2024-02-06T21:41:42.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7fafe97fa640 1 -- 172.21.15.120:0/1209179394 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe00052d0 con 0x7fafec070dc0 2024-02-06T21:41:42.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7fafe97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.538 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7fafe97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:42.538 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7fafe97fa640 10 monclient: dump: 2024-02-06T21:41:42.538 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.538 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.538 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.538 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.538 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.538 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.538 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.538 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.538 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.538 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.538 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7fafe97fa640 10 monclient: _finish_auth 0 2024-02-06T21:41:42.538 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7fafe97fa640 10 monclient: _check_auth_tickets 2024-02-06T21:41:42.538 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7fafe97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7fafe97fa640 1 -- 172.21.15.120:0/1209179394 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe0005430 con 0x7fafec070dc0 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7fafe97fa640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7fafe97fa640 1 -- 172.21.15.120:0/1209179394 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe0003830 con 0x7fafec070dc0 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7fafe97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7fafe97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7fafe97fa640 10 monclient: dump: 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.539 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7f8021f04640 1 -- 172.21.15.120:0/2504394258 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f800806d1a0 msgr2=0x7f800806f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7f8021f04640 1 --2- 172.21.15.120:0/2504394258 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f800806d1a0 0x7f800806f660 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f800c009710 tx=0x7f800c006c90 comp rx=0 tx=0).stop 2024-02-06T21:41:42.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7f8021f04640 10 monclient: shutdown 2024-02-06T21:41:42.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7f8021f04640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:42.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7f8021f04640 1 -- 172.21.15.120:0/2504394258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c142bb0 msgr2=0x7f801c13a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7f8021f04640 1 --2- 172.21.15.120:0/2504394258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c142bb0 0x7f801c13a490 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f801400c780 tx=0x7f801400cc40 comp rx=0 tx=0).stop 2024-02-06T21:41:42.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7fafeaffd640 4 set_mon_vals no callback set 2024-02-06T21:41:42.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7faff2aa1640 5 monclient: authenticate success, global_id 4381 2024-02-06T21:41:42.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7faff2aa1640 10 monclient: _renew_subs 2024-02-06T21:41:42.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7faff2aa1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:42.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.535+0000 7faff2aa1640 1 -- 172.21.15.120:0/1209179394 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafec11ef50 con 0x7fafec070dc0 2024-02-06T21:41:42.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7faff2aa1640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:42.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7faff2aa1640 10 monclient: _renew_subs 2024-02-06T21:41:42.544 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7faff2aa1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:42.544 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7faff2aa1640 1 -- 172.21.15.120:0/1209179394 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafec13f3b0 con 0x7fafec070dc0 2024-02-06T21:41:42.544 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7faff2aa1640 1 librados: init done 2024-02-06T21:41:42.544 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7f8021f04640 1 -- 172.21.15.120:0/2504394258 shutdown_connections 2024-02-06T21:41:42.544 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7f8021f04640 1 --2- 172.21.15.120:0/2504394258 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f800806d1a0 0x7f800806f660 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.544 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7f8021f04640 1 --2- 172.21.15.120:0/2504394258 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f801c13f810 0x7f801c139730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.545 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7f8021f04640 1 --2- 172.21.15.120:0/2504394258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f801c142bb0 0x7f801c13a490 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.545 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7f8021f04640 1 --2- 172.21.15.120:0/2504394258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f801c0a9c80 0x7f801c1417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.545 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7f8021f04640 1 -- 172.21.15.120:0/2504394258 >> 172.21.15.120:0/2504394258 conn(0x7f801c0b4ba0 msgr2=0x7f801c13b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:42.545 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7f8021f04640 1 -- 172.21.15.120:0/2504394258 shutdown_connections 2024-02-06T21:41:42.545 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.536+0000 7f8021f04640 1 -- 172.21.15.120:0/2504394258 wait complete. 2024-02-06T21:41:42.545 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.537+0000 7f8021f04640 1 librados: shutdown 2024-02-06T21:41:42.545 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.537+0000 7faff2aa1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.545 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.538+0000 7faff2aa1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.545 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.538+0000 7faff2aa1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:42.545 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.538+0000 7faff2aa1640 1 -- 172.21.15.120:0/1209179394 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafec144880 con 0x7fafec070dc0 2024-02-06T21:41:42.551 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444041 for osd.4 2024-02-06T21:41:42.551 DEBUG:teuthology.parallel:result is None 2024-02-06T21:41:42.626 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.625+0000 7fbda4806640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-02-06T21:41:42.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.625+0000 7fbda4806640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-02-06T21:41:42.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.625+0000 7fbda4806640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:42.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.625+0000 7fbda4806640 1 -- 172.21.15.120:0/1096688207 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fbda013ce10 con 0x7fbda00b54b0 2024-02-06T21:41:42.633 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-02-06T21:41:42.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.671+0000 7f1ca504a640 10 monclient: get_monmap_and_config 2024-02-06T21:41:42.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.671+0000 7f1ca504a640 10 monclient: build_initial_monmap 2024-02-06T21:41:42.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.671+0000 7f1ca504a640 10 monclient: monmap: 2024-02-06T21:41:42.673 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:42.673 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.673 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:42.673200+0000 2024-02-06T21:41:42.674 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:42.673200+0000 2024-02-06T21:41:42.674 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:42.674 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.674 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:42.674 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:42.674 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:42.674 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:42.674 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:42.674 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.675 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.672+0000 7f1ca504a640 1 Processor -- start 2024-02-06T21:41:42.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.672+0000 7f1ca504a640 1 -- start start 2024-02-06T21:41:42.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.672+0000 7f1ca504a640 10 monclient: init 2024-02-06T21:41:42.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.673+0000 7f1ca504a640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:42.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.673+0000 7f1ca504a640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-02-06T21:41:42.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.673+0000 7f1ca504a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1ca00b54b0 0x7f1ca00a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.673+0000 7f1ca504a640 10 monclient(hunting): picked mon.noname-a con 0x7f1ca00b54b0 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:42.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.673+0000 7f1ca504a640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1ca013f830 0x7f1ca013fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.673+0000 7f1ca504a640 10 monclient(hunting): picked mon.noname-c con 0x7f1ca013f830 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:42.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.673+0000 7f1ca504a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ca0142680 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:42.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.673+0000 7f1ca504a640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca0143200 con 0x7f1ca0142680 2024-02-06T21:41:42.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.673+0000 7f1ca504a640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.673+0000 7f1ca504a640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca00ab520 con 0x7f1ca00b54b0 2024-02-06T21:41:42.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.673+0000 7f1ca504a640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.673+0000 7f1ca504a640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca00b5890 con 0x7f1ca013f830 2024-02-06T21:41:42.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.673+0000 7f1ca504a640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:42.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.673+0000 7f1ca504a640 10 monclient(hunting): authenticate will time out at 1333.011022s 2024-02-06T21:41:42.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.674+0000 7f1c9ed9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1ca00b54b0 0x7f1ca00a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.674+0000 7f1c9ed9d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1ca00b54b0 0x7f1ca00a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:33348/0 (socket says 172.21.15.120:33348) 2024-02-06T21:41:42.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.674+0000 7f1c9ed9d640 1 -- 172.21.15.120:0/4035913574 learned_addr learned my addr 172.21.15.120:0/4035913574 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:42.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.674+0000 7f1c9ed9d640 10 monclient(hunting): get_auth_request con 0x7f1ca00b54b0 auth_method 0 2024-02-06T21:41:42.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.674+0000 7f1c9ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:42.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.674+0000 7f1c9ed9d640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:42.677 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.674+0000 7f1c9ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:42.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.674+0000 7f1c9ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:42.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.674+0000 7f1c9ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:42.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.674+0000 7f1c9ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:42.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.676+0000 7f1c9ed9d640 10 monclient(hunting): handle_auth_done global_id 4377 payload 274 2024-02-06T21:41:42.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.676+0000 7f1c9ed9d640 1 -- 172.21.15.120:0/4035913574 >> v2:172.21.15.135:3300/0 conn(0x7f1ca013f830 msgr2=0x7f1ca013fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:42.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.676+0000 7f1c9ed9d640 1 --2- 172.21.15.120:0/4035913574 >> v2:172.21.15.135:3300/0 conn(0x7f1ca013f830 0x7f1ca013fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.676+0000 7f1c9ed9d640 1 -- 172.21.15.120:0/4035913574 >> v1:172.21.15.135:6789/0 conn(0x7f1ca0142680 legacy=0x7f1ca0142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:42.678 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.676+0000 7f1c9ed9d640 10 monclient: _finish_hunting 0 2024-02-06T21:41:42.679 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.676+0000 7f1c9ed9d640 1 monclient: found mon.noname-a 2024-02-06T21:41:42.679 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.676+0000 7f1c9ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:42.679 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.676+0000 7f1c9ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:42.679 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.676+0000 7f1c9ed9d640 1 -- 172.21.15.120:0/4035913574 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca00ab720 con 0x7f1ca00b54b0 2024-02-06T21:41:42.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.679+0000 7f1c9ed9d640 1 --2- 172.21.15.120:0/4035913574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ca00b54b0 0x7f1ca00a9860 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1ca006deb0 tx=0x7f1c9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12bbdc608c136a43 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.685 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1c9dd9b640 1 -- 172.21.15.120:0/4035913574 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c90031870 con 0x7f1ca00b54b0 2024-02-06T21:41:42.685 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1c9dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.685 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1c9dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:42.685 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1c9dd9b640 10 monclient: dump: 2024-02-06T21:41:42.685 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.685 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.685 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.685 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.685 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.685 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.685 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.686 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.686 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.686 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1c9dd9b640 10 monclient: _finish_auth 0 2024-02-06T21:41:42.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1c9dd9b640 10 monclient: _check_auth_tickets 2024-02-06T21:41:42.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1c9dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:42.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1c9dd9b640 1 -- 172.21.15.120:0/4035913574 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c900319d0 con 0x7f1ca00b54b0 2024-02-06T21:41:42.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1c9dd9b640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:42.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1c9dd9b640 1 -- 172.21.15.120:0/4035913574 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c90031d60 con 0x7f1ca00b54b0 2024-02-06T21:41:42.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1c9dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1c9dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:42.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1c9dd9b640 10 monclient: dump: 2024-02-06T21:41:42.686 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.686 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.686 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.687 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.687 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.687 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.687 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.687 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.687 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.687 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1ca504a640 5 monclient: authenticate success, global_id 4377 2024-02-06T21:41:42.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1ca504a640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:42.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.683+0000 7f1ca504a640 4 set_mon_vals no callback set 2024-02-06T21:41:42.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.684+0000 7f1ca504a640 10 monclient: shutdown 2024-02-06T21:41:42.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.684+0000 7f1ca504a640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:42.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.684+0000 7f1ca504a640 1 -- 172.21.15.120:0/4035913574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ca00b54b0 msgr2=0x7f1ca00a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.684+0000 7f1ca504a640 1 --2- 172.21.15.120:0/4035913574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ca00b54b0 0x7f1ca00a9860 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1ca006deb0 tx=0x7f1c9002b910 comp rx=0 tx=0).stop 2024-02-06T21:41:42.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.684+0000 7f1ca504a640 1 -- 172.21.15.120:0/4035913574 shutdown_connections 2024-02-06T21:41:42.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.684+0000 7f1ca504a640 1 --2- 172.21.15.120:0/4035913574 >> v2:172.21.15.135:3300/0 conn(0x7f1ca013f830 0x7f1ca013fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.690 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.684+0000 7f1ca504a640 1 --2- 172.21.15.120:0/4035913574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ca00b54b0 0x7f1ca00a9860 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.691 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.684+0000 7f1ca504a640 1 -- 172.21.15.120:0/4035913574 >> 172.21.15.120:0/4035913574 conn(0x7f1ca00b8660 msgr2=0x7f1ca00b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:42.691 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.684+0000 7f1ca504a640 1 -- 172.21.15.120:0/4035913574 shutdown_connections 2024-02-06T21:41:42.691 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.684+0000 7f1ca504a640 1 -- 172.21.15.120:0/4035913574 wait complete. 2024-02-06T21:41:42.691 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.685+0000 7f1ca504a640 10 monclient: build_initial_monmap 2024-02-06T21:41:42.691 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.685+0000 7f1ca504a640 10 monclient: monmap: 2024-02-06T21:41:42.691 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:42.691 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:42.691 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:42.691 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:42.691 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:42.691 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.692 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:42.692 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:42.692 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:42.692 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.686+0000 7f1ca504a640 1 librados: starting msgr at 2024-02-06T21:41:42.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.686+0000 7f1ca504a640 1 librados: starting objecter 2024-02-06T21:41:42.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.686+0000 7f1ca504a640 1 Processor -- start 2024-02-06T21:41:42.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.686+0000 7f1ca504a640 1 -- start start 2024-02-06T21:41:42.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.686+0000 7f1ca504a640 1 librados: setting wanted keys 2024-02-06T21:41:42.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.686+0000 7f1ca504a640 1 librados: calling monclient init 2024-02-06T21:41:42.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.686+0000 7f1ca504a640 10 monclient: init 2024-02-06T21:41:42.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:42.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 10 monclient: _add_conns ranks=[2,0,1] 2024-02-06T21:41:42.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ca00b54b0 0x7f1ca0138ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 10 monclient(hunting): picked mon.noname-c con 0x7f1ca00b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:42.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ca013f830 0x7f1ca01320f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 10 monclient(hunting): picked mon.noname-a con 0x7f1ca013f830 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:42.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ca0142680 0x7f1ca0132630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 10 monclient(hunting): picked mon.noname-b con 0x7f1ca0142680 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:42.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca0139400 con 0x7f1ca013f830 2024-02-06T21:41:42.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca0137000 con 0x7f1ca0142680 2024-02-06T21:41:42.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca0138aa0 con 0x7f1ca00b54b0 2024-02-06T21:41:42.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:42.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1ca504a640 10 monclient(hunting): authenticate will time out at 1333.024664s 2024-02-06T21:41:42.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.687+0000 7f1c9e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ca013f830 0x7f1ca01320f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.688+0000 7f1c9e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ca013f830 0x7f1ca01320f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:33352/0 (socket says 172.21.15.120:33352) 2024-02-06T21:41:42.694 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.688+0000 7f1c9e59c640 1 -- 172.21.15.120:0/1127179225 learned_addr learned my addr 172.21.15.120:0/1127179225 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:42.694 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.688+0000 7f1c9e59c640 10 monclient(hunting): get_auth_request con 0x7f1ca013f830 auth_method 0 2024-02-06T21:41:42.694 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.688+0000 7f1c9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:42.694 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.688+0000 7f1c9e59c640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:42.694 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.688+0000 7f1c9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:42.694 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.688+0000 7f1c9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:42.694 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.688+0000 7f1c9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:42.694 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.688+0000 7f1c9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:42.695 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.691+0000 7f1c9e59c640 10 monclient(hunting): handle_auth_done global_id 4380 payload 824 2024-02-06T21:41:42.695 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.691+0000 7f1c9e59c640 1 -- 172.21.15.120:0/1127179225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ca00b54b0 msgr2=0x7f1ca0138ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.691+0000 7f1c9e59c640 1 --2- 172.21.15.120:0/1127179225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ca00b54b0 0x7f1ca0138ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.691+0000 7f1c9e59c640 1 -- 172.21.15.120:0/1127179225 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ca0142680 msgr2=0x7f1ca0132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.691+0000 7f1c9e59c640 1 --2- 172.21.15.120:0/1127179225 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ca0142680 0x7f1ca0132630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.691+0000 7f1c9e59c640 10 monclient: _finish_hunting 0 2024-02-06T21:41:42.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.691+0000 7f1c9e59c640 1 monclient: found mon.noname-a 2024-02-06T21:41:42.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.691+0000 7f1c9e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:42.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.691+0000 7f1c9e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:42.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.691+0000 7f1c9e59c640 1 -- 172.21.15.120:0/1127179225 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c900094d0 con 0x7f1ca013f830 2024-02-06T21:41:42.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.694+0000 7f1c9e59c640 1 --2- 172.21.15.120:0/1127179225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ca013f830 0x7f1ca01320f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400b4a0 tx=0x7f1c9400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.694+0000 7f1c877fe640 10 client.?.objecter ms_handle_connect 0x7f1ca013f830 2024-02-06T21:41:42.696 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.694+0000 7f1c877fe640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:42.700 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.697+0000 7f1c877fe640 1 -- 172.21.15.120:0/1127179225 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94010610 con 0x7f1ca013f830 2024-02-06T21:41:42.700 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.697+0000 7f1c877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.700 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.697+0000 7f1c877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:42.700 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.697+0000 7f1c877fe640 10 monclient: dump: 2024-02-06T21:41:42.700 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.700 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.700 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.700 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.697+0000 7f1c877fe640 10 monclient: _finish_auth 0 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.697+0000 7f1c877fe640 10 monclient: _check_auth_tickets 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.697+0000 7f1c877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.697+0000 7f1c877fe640 1 -- 172.21.15.120:0/1127179225 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c94010770 con 0x7f1ca013f830 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.697+0000 7f1c877fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.697+0000 7f1c877fe640 1 -- 172.21.15.120:0/1127179225 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94010b20 con 0x7f1ca013f830 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.698+0000 7f1c877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.698+0000 7f1c877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.698+0000 7f1c877fe640 10 monclient: dump: 2024-02-06T21:41:42.701 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.702 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.702 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.702 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.702 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.702 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.702 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.702 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.702 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.702 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.702 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.698+0000 7f1c9d59a640 4 set_mon_vals no callback set 2024-02-06T21:41:42.702 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.698+0000 7f1ca504a640 5 monclient: authenticate success, global_id 4380 2024-02-06T21:41:42.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.698+0000 7f1ca504a640 10 monclient: _renew_subs 2024-02-06T21:41:42.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.698+0000 7f1ca504a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:42.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.698+0000 7f1ca504a640 1 -- 172.21.15.120:0/1127179225 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca0132b70 con 0x7f1ca013f830 2024-02-06T21:41:42.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.698+0000 7f1ca504a640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:42.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.698+0000 7f1ca504a640 10 monclient: _renew_subs 2024-02-06T21:41:42.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.698+0000 7f1ca504a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:42.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.698+0000 7f1ca504a640 1 -- 172.21.15.120:0/1127179225 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca01377a0 con 0x7f1ca013f830 2024-02-06T21:41:42.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.698+0000 7f1ca504a640 1 librados: init done 2024-02-06T21:41:42.704 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.699+0000 7f1ca504a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.704 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.699+0000 7f1ca504a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.704 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.699+0000 7f1ca504a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:42.704 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.699+0000 7f1ca504a640 1 -- 172.21.15.120:0/1127179225 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca014ccc0 con 0x7f1ca013f830 2024-02-06T21:41:42.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.743+0000 7f751dffb640 1 -- 172.21.15.120:0/357746855 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f75200541b0 con 0x7f7528142680 2024-02-06T21:41:42.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.743+0000 7f751dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.743+0000 7f751dffb640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:42.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.774+0000 7f86abf0e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-02-06T21:41:42.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.774+0000 7f86abf0e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-02-06T21:41:42.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.774+0000 7f86abf0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:42.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.774+0000 7f86abf0e640 1 -- 172.21.15.120:0/967099549 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f86a41369a0 con 0x7f86a40aa1f0 2024-02-06T21:41:42.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.778+0000 7fafe97fa640 1 -- 172.21.15.120:0/1209179394 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fafe0003990 con 0x7fafec070dc0 2024-02-06T21:41:42.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.778+0000 7fe47008d640 10 monclient: get_monmap_and_config 2024-02-06T21:41:42.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.778+0000 7fe47008d640 10 monclient: build_initial_monmap 2024-02-06T21:41:42.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.778+0000 7fe47008d640 10 monclient: monmap: 2024-02-06T21:41:42.780 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:42.780 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.780 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:42.779794+0000 2024-02-06T21:41:42.780 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:42.779794+0000 2024-02-06T21:41:42.780 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:42.780 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.780 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:42.780 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:42.780 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:42.781 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:42.781 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:42.781 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.778+0000 7fafe97fa640 1 --2- 172.21.15.120:0/1209179394 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fafd006d1f0 0x7fafd006f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.778+0000 7fafebfff640 1 --2- 172.21.15.120:0/1209179394 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fafd006d1f0 0x7fafd006f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.778+0000 7fafebfff640 10 monclient: get_auth_request con 0x7fafd006d1f0 auth_method 0 2024-02-06T21:41:42.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.779+0000 7fe47008d640 1 Processor -- start 2024-02-06T21:41:42.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.779+0000 7fe47008d640 1 -- start start 2024-02-06T21:41:42.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.779+0000 7fe47008d640 10 monclient: init 2024-02-06T21:41:42.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.779+0000 7fafe97fa640 10 client.4381.objecter ms_handle_connect 0x7fafd006d1f0 2024-02-06T21:41:42.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.779+0000 7fafebfff640 1 --2- 172.21.15.120:0/1209179394 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fafd006d1f0 0x7fafd006f6b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fafec11ecc0 tx=0x7fafd8006c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.779+0000 7fe47008d640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:42.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.779+0000 7fe47008d640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-02-06T21:41:42.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.779+0000 7fe47008d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4680b54b0 0x7fe4680a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.779+0000 7fe47008d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4680b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:42.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.780+0000 7fe47008d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe46813f830 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:42.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.780+0000 7fe47008d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe468142680 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:42.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.780+0000 7fe47008d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe468143200 con 0x7fe468142680 2024-02-06T21:41:42.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.780+0000 7fe47008d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4680ab720 con 0x7fe46813f830 2024-02-06T21:41:42.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.780+0000 7fe47008d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.780+0000 7fe47008d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4680ab520 con 0x7fe4680b54b0 2024-02-06T21:41:42.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.780+0000 7fe47008d640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:42.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.780+0000 7fe47008d640 10 monclient(hunting): authenticate will time out at 1333.117682s 2024-02-06T21:41:42.785 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.782+0000 7fafe97fa640 1 -- 172.21.15.120:0/1209179394 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fafe0058ad0 con 0x7fafec070dc0 2024-02-06T21:41:42.785 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.782+0000 7fafe97fa640 10 client.4381.objecter ms_dispatch 0x7fafec000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:42.785 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.782+0000 7fafe97fa640 3 client.4381.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:42.785 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.782+0000 7fafe97fa640 3 client.4381.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:42.785 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.782+0000 7fafe97fa640 20 client.4381.objecter dump_active .. 0 homeless 2024-02-06T21:41:42.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.792+0000 7fafe97fa640 1 -- 172.21.15.120:0/1209179394 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fafe0054670 con 0x7fafec070dc0 2024-02-06T21:41:42.794 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.792+0000 7fafe97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.794 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.792+0000 7fafe97fa640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:42.800 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.798+0000 7fbd8effd640 1 -- 172.21.15.120:0/1096688207 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbd90035020 con 0x7fbda00b54b0 2024-02-06T21:41:42.800 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.798+0000 7fbd8effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-02-06T21:41:42.800 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.798+0000 7fbd8effd640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:77309411336 2024-02-06T21:41:42.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.801+0000 7fbda4806640 10 librados: watch_flush enter 2024-02-06T21:41:42.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.801+0000 7fbda4806640 10 librados: watch_flush exit 2024-02-06T21:41:42.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.801+0000 7fbda4806640 20 client.4375.objecter shutdown clearing up homeless session... 2024-02-06T21:41:42.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.801+0000 7fbda4806640 10 client.4375.objecter successfully canceled tick 2024-02-06T21:41:42.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.801+0000 7fbda4806640 1 -- 172.21.15.120:0/1096688207 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbd8006d1d0 msgr2=0x7fbd8006f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.801+0000 7fbda4806640 1 --2- 172.21.15.120:0/1096688207 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbd8006d1d0 0x7fbd8006f690 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbda013d0c0 tx=0x7fbd94006c90 comp rx=0 tx=0).stop 2024-02-06T21:41:42.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.801+0000 7fbda4806640 10 monclient: shutdown 2024-02-06T21:41:42.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.801+0000 7fbda4806640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:42.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.801+0000 7fbda4806640 1 -- 172.21.15.120:0/1096688207 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda00b54b0 msgr2=0x7fbda00b7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.801+0000 7fbda4806640 1 --2- 172.21.15.120:0/1096688207 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda00b54b0 0x7fbda00b7140 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbd90000c00 tx=0x7fbd900046c0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.802+0000 7fbda4806640 1 -- 172.21.15.120:0/1096688207 shutdown_connections 2024-02-06T21:41:42.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.802+0000 7fbda4806640 1 --2- 172.21.15.120:0/1096688207 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbd8006d1d0 0x7fbd8006f690 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.802+0000 7fbda4806640 1 --2- 172.21.15.120:0/1096688207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbda0142680 0x7fbda01d7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.802+0000 7fbda4806640 1 --2- 172.21.15.120:0/1096688207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbda013f830 0x7fbda00b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.802+0000 7fbda4806640 1 --2- 172.21.15.120:0/1096688207 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbda00b54b0 0x7fbda00b7140 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.802+0000 7fbda4806640 1 -- 172.21.15.120:0/1096688207 >> 172.21.15.120:0/1096688207 conn(0x7fbda00b8660 msgr2=0x7fbda00a9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:42.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.802+0000 7fbda4806640 1 -- 172.21.15.120:0/1096688207 shutdown_connections 2024-02-06T21:41:42.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.802+0000 7fbda4806640 1 -- 172.21.15.120:0/1096688207 wait complete. 2024-02-06T21:41:42.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.802+0000 7fbda4806640 1 librados: shutdown 2024-02-06T21:41:42.806 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.802+0000 7fe46d628640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fe46813f830 0x7fe46813fc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.120:53584/0 (socket says 172.21.15.120:53584) 2024-02-06T21:41:42.806 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.802+0000 7fe46d628640 1 -- 172.21.15.120:0/4283272132 learned_addr learned my addr 172.21.15.120:0/4283272132 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:41:42.818 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411336 got 77309411336 for osd.1 2024-02-06T21:41:42.819 DEBUG:teuthology.parallel:result is None 2024-02-06T21:41:42.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.842+0000 7fe46de29640 1 --2- 172.21.15.120:0/4283272132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4680b54b0 0x7fe4680a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.845 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.843+0000 7fe46de29640 10 monclient(hunting): get_auth_request con 0x7fe4680b54b0 auth_method 0 2024-02-06T21:41:42.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.843+0000 7fe46de29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:42.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.843+0000 7fe46de29640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:42.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.843+0000 7fe46de29640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:42.902 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.845+0000 7fe46de29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:42.902 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.845+0000 7fe46de29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:42.902 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.845+0000 7fe46de29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:42.902 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.847+0000 7fe46de29640 10 monclient(hunting): handle_auth_done global_id 4337 payload 274 2024-02-06T21:41:42.902 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.847+0000 7fe46de29640 1 -- 172.21.15.120:0/4283272132 >> v1:172.21.15.135:6789/0 conn(0x7fe46813f830 legacy=0x7fe46813fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.902 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.847+0000 7fe46de29640 1 -- 172.21.15.120:0/4283272132 >> v1:172.21.15.120:6789/0 conn(0x7fe468142680 legacy=0x7fe468142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:42.903 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.847+0000 7fe46de29640 10 monclient: _finish_hunting 0 2024-02-06T21:41:42.903 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.847+0000 7fe46de29640 1 monclient: found mon.noname-b 2024-02-06T21:41:42.903 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.847+0000 7fe46de29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:42.903 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.847+0000 7fe46de29640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T21:41:42.903 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.847+0000 7fe46de29640 1 -- 172.21.15.120:0/4283272132 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4680b4800 con 0x7fe4680b54b0 2024-02-06T21:41:42.903 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.850+0000 7fe46de29640 1 --2- 172.21.15.120:0/4283272132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4680b54b0 0x7fe4680a9860 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe4640097d0 tx=0x7fe46402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78fc03316e7d0527 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.903 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe46ce27640 1 -- 172.21.15.120:0/4283272132 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464031800 con 0x7fe4680b54b0 2024-02-06T21:41:42.903 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe46ce27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.903 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe46ce27640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:42.903 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe46ce27640 10 monclient: dump: 2024-02-06T21:41:42.903 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.903 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.903 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.903 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.904 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe46ce27640 10 monclient: _finish_auth 0 2024-02-06T21:41:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe46ce27640 10 monclient: _check_auth_tickets 2024-02-06T21:41:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe46ce27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe46ce27640 1 -- 172.21.15.120:0/4283272132 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe464031960 con 0x7fe4680b54b0 2024-02-06T21:41:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe46ce27640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe46ce27640 1 -- 172.21.15.120:0/4283272132 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464031cf0 con 0x7fe4680b54b0 2024-02-06T21:41:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe46ce27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe46ce27640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe46ce27640 10 monclient: dump: 2024-02-06T21:41:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.905 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe47008d640 5 monclient: authenticate success, global_id 4337 2024-02-06T21:41:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe47008d640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.855+0000 7fe47008d640 4 set_mon_vals no callback set 2024-02-06T21:41:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.856+0000 7fe47008d640 10 monclient: shutdown 2024-02-06T21:41:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.856+0000 7fe47008d640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.856+0000 7fe47008d640 1 -- 172.21.15.120:0/4283272132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4680b54b0 msgr2=0x7fe4680a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.856+0000 7fe47008d640 1 --2- 172.21.15.120:0/4283272132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4680b54b0 0x7fe4680a9860 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe4640097d0 tx=0x7fe46402b8a0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.856+0000 7fe47008d640 1 -- 172.21.15.120:0/4283272132 shutdown_connections 2024-02-06T21:41:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.856+0000 7fe47008d640 1 --2- 172.21.15.120:0/4283272132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4680b54b0 0x7fe4680a9860 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.856+0000 7fe47008d640 1 -- 172.21.15.120:0/4283272132 >> 172.21.15.120:0/4283272132 conn(0x7fe4680b8660 msgr2=0x7fe4680b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.856+0000 7fe47008d640 1 -- 172.21.15.120:0/4283272132 shutdown_connections 2024-02-06T21:41:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.856+0000 7fe47008d640 1 -- 172.21.15.120:0/4283272132 wait complete. 2024-02-06T21:41:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.857+0000 7fe47008d640 10 monclient: build_initial_monmap 2024-02-06T21:41:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.857+0000 7fe47008d640 10 monclient: monmap: 2024-02-06T21:41:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.857+0000 7fe47008d640 1 librados: starting msgr at 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.857+0000 7fe47008d640 1 librados: starting objecter 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.857+0000 7fe47008d640 1 Processor -- start 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.857+0000 7fe47008d640 1 -- start start 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.857+0000 7fe47008d640 1 librados: setting wanted keys 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.857+0000 7fe47008d640 1 librados: calling monclient init 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.857+0000 7fe47008d640 10 monclient: init 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.858+0000 7fe47008d640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.858+0000 7fe47008d640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.858+0000 7fe47008d640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4680b54b0 0x7fe468134b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.858+0000 7fe47008d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4680b54b0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.858+0000 7fe47008d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46813f830 0x7fe4681376c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.858+0000 7fe47008d640 10 monclient(hunting): picked mon.noname-c con 0x7fe46813f830 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.858+0000 7fe47008d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe468142680 0x7fe468137c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.858+0000 7fe47008d640 10 monclient(hunting): picked mon.noname-a con 0x7fe468142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.858+0000 7fe47008d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.858+0000 7fe47008d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe468136d70 con 0x7fe468142680 2024-02-06T21:41:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.858+0000 7fe47008d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.858+0000 7fe47008d640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4681354c0 con 0x7fe4680b54b0 2024-02-06T21:41:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.859+0000 7fe47008d640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.859+0000 7fe47008d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe468136fc0 con 0x7fe46813f830 2024-02-06T21:41:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.859+0000 7fe47008d640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.859+0000 7fe47008d640 10 monclient(hunting): authenticate will time out at 1333.196546s 2024-02-06T21:41:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.859+0000 7fe46d628640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46813f830 0x7fe4681376c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.859+0000 7fe46d628640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46813f830 0x7fe4681376c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:49216/0 (socket says 172.21.15.120:49216) 2024-02-06T21:41:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.859+0000 7fe46d628640 1 -- 172.21.15.120:0/2323686672 learned_addr learned my addr 172.21.15.120:0/2323686672 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.859+0000 7fe46d628640 10 monclient(hunting): get_auth_request con 0x7fe46813f830 auth_method 0 2024-02-06T21:41:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.859+0000 7fe46d628640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.859+0000 7fe46d628640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.859+0000 7fe46d628640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.860+0000 7fe46d628640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.860+0000 7fe46d628640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.860+0000 7fe46d628640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.864+0000 7fe46d628640 10 monclient(hunting): handle_auth_done global_id 4340 payload 824 2024-02-06T21:41:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.864+0000 7fe46d628640 1 -- 172.21.15.120:0/2323686672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4680b54b0 msgr2=0x7fe468134b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.864+0000 7fe46d628640 1 --2- 172.21.15.120:0/2323686672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4680b54b0 0x7fe468134b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.864+0000 7fe46d628640 1 -- 172.21.15.120:0/2323686672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe468142680 msgr2=0x7fe468137c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.864+0000 7fe46d628640 1 --2- 172.21.15.120:0/2323686672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe468142680 0x7fe468137c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.864+0000 7fe46d628640 10 monclient: _finish_hunting 0 2024-02-06T21:41:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.864+0000 7fe46d628640 1 monclient: found mon.noname-c 2024-02-06T21:41:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.864+0000 7fe46d628640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.864+0000 7fe46d628640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.864+0000 7fe46d628640 1 -- 172.21.15.120:0/2323686672 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe464009400 con 0x7fe46813f830 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.868+0000 7fe46d628640 1 --2- 172.21.15.120:0/2323686672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46813f830 0x7fe4681376c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe45800c430 tx=0x7fe45800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.868+0000 7fe45e7fc640 10 client.?.objecter ms_handle_connect 0x7fe46813f830 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.868+0000 7fe45e7fc640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe45e7fc640 1 -- 172.21.15.120:0/2323686672 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe458005ce0 con 0x7fe46813f830 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe45e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe45e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe45e7fc640 10 monclient: dump: 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.911 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.911 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.911 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe45e7fc640 10 monclient: _finish_auth 0 2024-02-06T21:41:42.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe45e7fc640 10 monclient: _check_auth_tickets 2024-02-06T21:41:42.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe45e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:42.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe45e7fc640 1 -- 172.21.15.120:0/2323686672 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe458005e40 con 0x7fe46813f830 2024-02-06T21:41:42.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe47008d640 5 monclient: authenticate success, global_id 4340 2024-02-06T21:41:42.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe47008d640 10 monclient: _renew_subs 2024-02-06T21:41:42.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe47008d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:42.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe47008d640 1 -- 172.21.15.120:0/2323686672 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4681372a0 con 0x7fe46813f830 2024-02-06T21:41:42.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe45e7fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:42.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe45e7fc640 1 -- 172.21.15.120:0/2323686672 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4580072f0 con 0x7fe46813f830 2024-02-06T21:41:42.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe45e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:42.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe45e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T21:41:42.912 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe45e7fc640 10 monclient: dump: 2024-02-06T21:41:42.912 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:42.912 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:42.912 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.912 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:42.912 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:42.912 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:42.912 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:42.912 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:42.912 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:42.912 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:42.913 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe47008d640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:42.913 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe45ffff640 4 set_mon_vals no callback set 2024-02-06T21:41:42.913 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe47008d640 10 monclient: _renew_subs 2024-02-06T21:41:42.913 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe47008d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:42.913 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe47008d640 1 -- 172.21.15.120:0/2323686672 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe468135df0 con 0x7fe46813f830 2024-02-06T21:41:42.913 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.873+0000 7fe47008d640 1 librados: init done 2024-02-06T21:41:42.913 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.874+0000 7fe47008d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.913 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.874+0000 7fe47008d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:42.913 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.874+0000 7fe47008d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:42.913 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.874+0000 7fe47008d640 1 -- 172.21.15.120:0/2323686672 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe468136010 con 0x7fe46813f830 2024-02-06T21:41:42.964 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.963+0000 7f752f63e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-02-06T21:41:42.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.963+0000 7f752f63e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-02-06T21:41:42.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.963+0000 7f752f63e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:42.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:42.963+0000 7f752f63e640 1 -- 172.21.15.120:0/357746855 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f752814df40 con 0x7f7528142680 2024-02-06T21:41:43.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.001+0000 7f1c877fe640 1 -- 172.21.15.120:0/1127179225 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1c94010c80 con 0x7f1ca013f830 2024-02-06T21:41:43.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.002+0000 7f1c877fe640 1 --2- 172.21.15.120:0/1127179225 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1c8c06d1f0 0x7f1c8c06f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:43.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.002+0000 7f1c9ed9d640 1 --2- 172.21.15.120:0/1127179225 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1c8c06d1f0 0x7f1c8c06f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:43.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.002+0000 7f1c9ed9d640 10 monclient: get_auth_request con 0x7f1c8c06d1f0 auth_method 0 2024-02-06T21:41:43.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.003+0000 7f1c877fe640 10 client.4380.objecter ms_handle_connect 0x7f1c8c06d1f0 2024-02-06T21:41:43.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.003+0000 7f1c9ed9d640 1 --2- 172.21.15.120:0/1127179225 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1c8c06d1f0 0x7f1c8c06f6b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1c90036000 tx=0x7f1c90030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:43.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.003+0000 7faff2aa1640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-02-06T21:41:43.005 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.003+0000 7faff2aa1640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-02-06T21:41:43.005 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.003+0000 7faff2aa1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:43.005 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.003+0000 7faff2aa1640 1 -- 172.21.15.120:0/1209179394 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fafec04ef90 con 0x7fafec070dc0 2024-02-06T21:41:43.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.006+0000 7fafe97fa640 1 -- 172.21.15.120:0/1209179394 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fafe0017020 con 0x7fafec070dc0 2024-02-06T21:41:43.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.006+0000 7fafe97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-02-06T21:41:43.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.006+0000 7fafe97fa640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:43.008 INFO:teuthology.orchestra.run.smithi120.stdout:73014444041 2024-02-06T21:41:43.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.008+0000 7faff2aa1640 10 librados: watch_flush enter 2024-02-06T21:41:43.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.008+0000 7faff2aa1640 10 librados: watch_flush exit 2024-02-06T21:41:43.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.008+0000 7faff2aa1640 20 client.4381.objecter shutdown clearing up homeless session... 2024-02-06T21:41:43.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.008+0000 7faff2aa1640 10 client.4381.objecter successfully canceled tick 2024-02-06T21:41:43.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.008+0000 7faff2aa1640 1 -- 172.21.15.120:0/1209179394 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fafd006d1f0 msgr2=0x7fafd006f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:43.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.008+0000 7faff2aa1640 1 --2- 172.21.15.120:0/1209179394 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fafd006d1f0 0x7fafd006f6b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fafec11ecc0 tx=0x7fafd8006c90 comp rx=0 tx=0).stop 2024-02-06T21:41:43.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.008+0000 7faff2aa1640 10 monclient: shutdown 2024-02-06T21:41:43.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.008+0000 7faff2aa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:43.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.008+0000 7faff2aa1640 1 -- 172.21.15.120:0/1209179394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec070dc0 msgr2=0x7fafec0ac140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:43.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.008+0000 7faff2aa1640 1 --2- 172.21.15.120:0/1209179394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec070dc0 0x7fafec0ac140 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fafe000c400 tx=0x7fafe000c8c0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.009+0000 7faff2aa1640 1 -- 172.21.15.120:0/1209179394 shutdown_connections 2024-02-06T21:41:43.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.009+0000 7faff2aa1640 1 --2- 172.21.15.120:0/1209179394 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fafd006d1f0 0x7fafd006f6b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.009+0000 7faff2aa1640 1 --2- 172.21.15.120:0/1209179394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafec0ad7b0 0x7fafec13ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.009+0000 7faff2aa1640 1 --2- 172.21.15.120:0/1209179394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafec0711a0 0x7fafec1d58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.009+0000 7faff2aa1640 1 --2- 172.21.15.120:0/1209179394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec070dc0 0x7fafec0ac140 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.009+0000 7faff2aa1640 1 -- 172.21.15.120:0/1209179394 >> 172.21.15.120:0/1209179394 conn(0x7fafec0b6250 msgr2=0x7fafec0b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:43.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.009+0000 7faff2aa1640 1 -- 172.21.15.120:0/1209179394 shutdown_connections 2024-02-06T21:41:43.012 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.009+0000 7faff2aa1640 1 -- 172.21.15.120:0/1209179394 wait complete. 2024-02-06T21:41:43.012 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.009+0000 7faff2aa1640 1 librados: shutdown 2024-02-06T21:41:43.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.016+0000 7f1c877fe640 1 -- 172.21.15.120:0/1127179225 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1c940590d0 con 0x7f1ca013f830 2024-02-06T21:41:43.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.016+0000 7f1c877fe640 10 client.4380.objecter ms_dispatch 0x7f1ca0000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:43.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.016+0000 7f1c877fe640 3 client.4380.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:43.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.016+0000 7f1c877fe640 3 client.4380.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:43.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.017+0000 7f1c877fe640 20 client.4380.objecter dump_active .. 0 homeless 2024-02-06T21:41:43.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.017+0000 7f1c877fe640 1 -- 172.21.15.120:0/1127179225 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1c94054c70 con 0x7f1ca013f830 2024-02-06T21:41:43.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.017+0000 7f1c877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:43.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.017+0000 7f1c877fe640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:43.021 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444041 for osd.6 2024-02-06T21:41:43.021 DEBUG:teuthology.parallel:result is None 2024-02-06T21:41:43.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.028+0000 7f869a7fc640 1 -- 172.21.15.120:0/967099549 <== mon.0 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f86a41369a0 con 0x7f86a40aa1f0 2024-02-06T21:41:43.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.028+0000 7f869a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-02-06T21:41:43.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.028+0000 7f869a7fc640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:43.030 INFO:teuthology.orchestra.run.smithi120.stdout:55834574854 2024-02-06T21:41:43.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 10 librados: watch_flush enter 2024-02-06T21:41:43.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 10 librados: watch_flush exit 2024-02-06T21:41:43.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 20 client.4374.objecter shutdown clearing up homeless session... 2024-02-06T21:41:43.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 10 client.4374.objecter successfully canceled tick 2024-02-06T21:41:43.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 1 -- 172.21.15.120:0/967099549 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f868806d220 msgr2=0x7f868806f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:43.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 1 --2- 172.21.15.120:0/967099549 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f868806d220 0x7f868806f6e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f86a0001560 tx=0x7f86a0007990 comp rx=0 tx=0).stop 2024-02-06T21:41:43.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 10 monclient: shutdown 2024-02-06T21:41:43.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:43.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 1 -- 172.21.15.120:0/967099549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a40aa1f0 msgr2=0x7f86a40abd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:43.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 1 --2- 172.21.15.120:0/967099549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a40aa1f0 0x7f86a40abd90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f869c00b9a0 tx=0x7f869c00be60 comp rx=0 tx=0).stop 2024-02-06T21:41:43.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 1 -- 172.21.15.120:0/967099549 shutdown_connections 2024-02-06T21:41:43.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 1 --2- 172.21.15.120:0/967099549 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f868806d220 0x7f868806f6e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 1 --2- 172.21.15.120:0/967099549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86a40aa1f0 0x7f86a40abd90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 1 --2- 172.21.15.120:0/967099549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86a40ae8e0 0x7f86a40ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 1 --2- 172.21.15.120:0/967099549 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f86a4072540 0x7f86a40a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 1 -- 172.21.15.120:0/967099549 >> 172.21.15.120:0/967099549 conn(0x7f86a40b0bc0 msgr2=0x7f86a40af9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:43.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 1 -- 172.21.15.120:0/967099549 shutdown_connections 2024-02-06T21:41:43.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 1 -- 172.21.15.120:0/967099549 wait complete. 2024-02-06T21:41:43.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.030+0000 7f86abf0e640 1 librados: shutdown 2024-02-06T21:41:43.043 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.0 2024-02-06T21:41:43.044 DEBUG:teuthology.parallel:result is None 2024-02-06T21:41:43.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.181+0000 7fe45e7fc640 1 -- 172.21.15.120:0/2323686672 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe458007510 con 0x7fe46813f830 2024-02-06T21:41:43.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.182+0000 7fe45e7fc640 1 --2- 172.21.15.120:0/2323686672 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe44c06d1a0 0x7fe44c06f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:43.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.182+0000 7fe46de29640 1 --2- 172.21.15.120:0/2323686672 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe44c06d1a0 0x7fe44c06f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:43.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.182+0000 7fe46de29640 10 monclient: get_auth_request con 0x7fe44c06d1a0 auth_method 0 2024-02-06T21:41:43.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.183+0000 7fe46de29640 1 --2- 172.21.15.120:0/2323686672 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe44c06d1a0 0x7fe44c06f660 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe464036000 tx=0x7fe464030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:43.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.183+0000 7fe45e7fc640 10 client.4340.objecter ms_handle_connect 0x7fe44c06d1a0 2024-02-06T21:41:43.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.198+0000 7fe45e7fc640 1 -- 172.21.15.120:0/2323686672 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe458058b30 con 0x7fe46813f830 2024-02-06T21:41:43.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.198+0000 7fe45e7fc640 10 client.4340.objecter ms_dispatch 0x7fe468000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:43.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.198+0000 7fe45e7fc640 3 client.4340.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:43.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.198+0000 7fe45e7fc640 3 client.4340.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:43.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.198+0000 7fe45e7fc640 20 client.4340.objecter dump_active .. 0 homeless 2024-02-06T21:41:43.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.198+0000 7fe45e7fc640 1 -- 172.21.15.120:0/2323686672 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe4580546d0 con 0x7fe46813f830 2024-02-06T21:41:43.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.198+0000 7fe45e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:43.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.198+0000 7fe45e7fc640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:43.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.200+0000 7f751dffb640 1 -- 172.21.15.120:0/357746855 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f752814df40 con 0x7f7528142680 2024-02-06T21:41:43.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.200+0000 7f751dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-02-06T21:41:43.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.200+0000 7f751dffb640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:43.201 INFO:teuthology.orchestra.run.smithi120.stdout:73014444041 2024-02-06T21:41:43.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 10 librados: watch_flush enter 2024-02-06T21:41:43.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 10 librados: watch_flush exit 2024-02-06T21:41:43.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 20 client.4334.objecter shutdown clearing up homeless session... 2024-02-06T21:41:43.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 10 client.4334.objecter successfully canceled tick 2024-02-06T21:41:43.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 1 -- 172.21.15.120:0/357746855 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f750c06d1a0 msgr2=0x7f750c06f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:43.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 1 --2- 172.21.15.120:0/357746855 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f750c06d1a0 0x7f750c06f660 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7524009650 tx=0x7f7524006ba0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 10 monclient: shutdown 2024-02-06T21:41:43.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:43.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 1 -- 172.21.15.120:0/357746855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7528142680 msgr2=0x7f75281375f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:43.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 1 --2- 172.21.15.120:0/357746855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7528142680 0x7f75281375f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f752000c7b0 tx=0x7f752000cc70 comp rx=0 tx=0).stop 2024-02-06T21:41:43.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 1 -- 172.21.15.120:0/357746855 shutdown_connections 2024-02-06T21:41:43.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 1 --2- 172.21.15.120:0/357746855 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f750c06d1a0 0x7f750c06f660 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 1 --2- 172.21.15.120:0/357746855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7528142680 0x7f75281375f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 1 --2- 172.21.15.120:0/357746855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f752813ff40 0x7f75281353c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 1 --2- 172.21.15.120:0/357746855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75280b54b0 0x7f75280b7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 1 -- 172.21.15.120:0/357746855 >> 172.21.15.120:0/357746855 conn(0x7f75280b8660 msgr2=0x7f752813f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:43.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 1 -- 172.21.15.120:0/357746855 shutdown_connections 2024-02-06T21:41:43.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 1 -- 172.21.15.120:0/357746855 wait complete. 2024-02-06T21:41:43.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.202+0000 7f752f63e640 1 librados: shutdown 2024-02-06T21:41:43.219 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444041 for osd.3 2024-02-06T21:41:43.219 DEBUG:teuthology.parallel:result is None 2024-02-06T21:41:43.246 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.244+0000 7f1ca504a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-02-06T21:41:43.246 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.244+0000 7f1ca504a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-02-06T21:41:43.246 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.244+0000 7f1ca504a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:43.246 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.244+0000 7f1ca504a640 1 -- 172.21.15.120:0/1127179225 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f1ca004ef90 con 0x7f1ca013f830 2024-02-06T21:41:43.251 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.250+0000 7f1c877fe640 1 -- 172.21.15.120:0/1127179225 <== mon.0 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1c9401e070 con 0x7f1ca013f830 2024-02-06T21:41:43.252 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.250+0000 7f1c877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-02-06T21:41:43.252 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.250+0000 7f1c877fe640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:43.252 INFO:teuthology.orchestra.run.smithi120.stdout:55834574855 2024-02-06T21:41:43.255 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.252+0000 7f1ca504a640 10 librados: watch_flush enter 2024-02-06T21:41:43.255 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.252+0000 7f1ca504a640 10 librados: watch_flush exit 2024-02-06T21:41:43.255 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.252+0000 7f1ca504a640 20 client.4380.objecter shutdown clearing up homeless session... 2024-02-06T21:41:43.255 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.252+0000 7f1ca504a640 10 client.4380.objecter successfully canceled tick 2024-02-06T21:41:43.255 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.253+0000 7f1ca504a640 1 -- 172.21.15.120:0/1127179225 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1c8c06d1f0 msgr2=0x7f1c8c06f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:43.255 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.253+0000 7f1ca504a640 1 --2- 172.21.15.120:0/1127179225 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1c8c06d1f0 0x7f1c8c06f6b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1c90036000 tx=0x7f1c90030000 comp rx=0 tx=0).stop 2024-02-06T21:41:43.255 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.253+0000 7f1ca504a640 10 monclient: shutdown 2024-02-06T21:41:43.255 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.253+0000 7f1ca504a640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:43.255 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.253+0000 7f1ca504a640 1 -- 172.21.15.120:0/1127179225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ca013f830 msgr2=0x7f1ca01320f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:43.255 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.253+0000 7f1ca504a640 1 --2- 172.21.15.120:0/1127179225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ca013f830 0x7f1ca01320f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400b4a0 tx=0x7f1c9400b960 comp rx=0 tx=0).stop 2024-02-06T21:41:43.255 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.253+0000 7f1ca504a640 1 -- 172.21.15.120:0/1127179225 shutdown_connections 2024-02-06T21:41:43.255 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.253+0000 7f1ca504a640 1 --2- 172.21.15.120:0/1127179225 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1c8c06d1f0 0x7f1c8c06f6b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.256 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.253+0000 7f1ca504a640 1 --2- 172.21.15.120:0/1127179225 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ca0142680 0x7f1ca0132630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.256 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.253+0000 7f1ca504a640 1 --2- 172.21.15.120:0/1127179225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ca013f830 0x7f1ca01320f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.256 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.253+0000 7f1ca504a640 1 --2- 172.21.15.120:0/1127179225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ca00b54b0 0x7f1ca0138ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.256 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.253+0000 7f1ca504a640 1 -- 172.21.15.120:0/1127179225 >> 172.21.15.120:0/1127179225 conn(0x7f1ca00b8660 msgr2=0x7f1ca00744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:43.256 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.253+0000 7f1ca504a640 1 -- 172.21.15.120:0/1127179225 shutdown_connections 2024-02-06T21:41:43.256 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.253+0000 7f1ca504a640 1 -- 172.21.15.120:0/1127179225 wait complete. 2024-02-06T21:41:43.256 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.253+0000 7f1ca504a640 1 librados: shutdown 2024-02-06T21:41:43.267 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.7 2024-02-06T21:41:43.267 DEBUG:teuthology.parallel:result is None 2024-02-06T21:41:43.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.394+0000 7fe47008d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-02-06T21:41:43.395 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.394+0000 7fe47008d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-02-06T21:41:43.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.394+0000 7fe47008d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T21:41:43.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.394+0000 7fe47008d640 1 -- 172.21.15.120:0/2323686672 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe46804ef90 con 0x7fe46813f830 2024-02-06T21:41:43.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.397+0000 7fe45e7fc640 1 -- 172.21.15.120:0/2323686672 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe458017070 con 0x7fe46813f830 2024-02-06T21:41:43.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.397+0000 7fe45e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-02-06T21:41:43.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.397+0000 7fe45e7fc640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:43.399 INFO:teuthology.orchestra.run.smithi120.stdout:73014444041 2024-02-06T21:41:43.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.399+0000 7fe47008d640 10 librados: watch_flush enter 2024-02-06T21:41:43.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.399+0000 7fe47008d640 10 librados: watch_flush exit 2024-02-06T21:41:43.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.399+0000 7fe47008d640 20 client.4340.objecter shutdown clearing up homeless session... 2024-02-06T21:41:43.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.399+0000 7fe47008d640 10 client.4340.objecter successfully canceled tick 2024-02-06T21:41:43.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.400+0000 7fe47008d640 1 -- 172.21.15.120:0/2323686672 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe44c06d1a0 msgr2=0x7fe44c06f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:43.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.400+0000 7fe47008d640 1 --2- 172.21.15.120:0/2323686672 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe44c06d1a0 0x7fe44c06f660 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe464036000 tx=0x7fe464030000 comp rx=0 tx=0).stop 2024-02-06T21:41:43.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.400+0000 7fe47008d640 10 monclient: shutdown 2024-02-06T21:41:43.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.400+0000 7fe47008d640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:43.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.400+0000 7fe47008d640 1 -- 172.21.15.120:0/2323686672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46813f830 msgr2=0x7fe4681376c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:43.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.400+0000 7fe47008d640 1 --2- 172.21.15.120:0/2323686672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46813f830 0x7fe4681376c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe45800c430 tx=0x7fe45800c8f0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.400+0000 7fe47008d640 1 -- 172.21.15.120:0/2323686672 shutdown_connections 2024-02-06T21:41:43.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.400+0000 7fe47008d640 1 --2- 172.21.15.120:0/2323686672 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe44c06d1a0 0x7fe44c06f660 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.400+0000 7fe47008d640 1 --2- 172.21.15.120:0/2323686672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe468142680 0x7fe468137c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.400+0000 7fe47008d640 1 --2- 172.21.15.120:0/2323686672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46813f830 0x7fe4681376c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.400+0000 7fe47008d640 1 --2- 172.21.15.120:0/2323686672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4680b54b0 0x7fe468134b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.400+0000 7fe47008d640 1 -- 172.21.15.120:0/2323686672 >> 172.21.15.120:0/2323686672 conn(0x7fe4680b8660 msgr2=0x7fe4680744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:43.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.400+0000 7fe47008d640 1 -- 172.21.15.120:0/2323686672 shutdown_connections 2024-02-06T21:41:43.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.400+0000 7fe47008d640 1 -- 172.21.15.120:0/2323686672 wait complete. 2024-02-06T21:41:43.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.400+0000 7fe47008d640 1 librados: shutdown 2024-02-06T21:41:43.413 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444041 for osd.2 2024-02-06T21:41:43.413 DEBUG:teuthology.parallel:result is None 2024-02-06T21:41:43.413 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-02-06T21:41:43.414 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-06T21:41:43.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.513+0000 7ff0ed80f640 10 monclient: get_monmap_and_config 2024-02-06T21:41:43.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.513+0000 7ff0ed80f640 10 monclient: build_initial_monmap 2024-02-06T21:41:43.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.513+0000 7ff0ed80f640 10 monclient: monmap: 2024-02-06T21:41:43.515 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:43.515 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:43.515 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:43.514986+0000 2024-02-06T21:41:43.515 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:43.514986+0000 2024-02-06T21:41:43.515 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:43.516 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:43.516 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:43.516 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:43.516 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:43.516 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:43.516 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:43.516 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:43.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 1 Processor -- start 2024-02-06T21:41:43.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 1 -- start start 2024-02-06T21:41:43.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 10 monclient: init 2024-02-06T21:41:43.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:43.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-02-06T21:41:43.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e80a9c80 0x7ff0e80aa060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:43.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 10 monclient(hunting): picked mon.noname-b con 0x7ff0e80a9c80 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:43.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff0e8142bb0 0x7ff0e8142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:43.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 10 monclient(hunting): picked mon.noname-c con 0x7ff0e8142bb0 addr v2:172.21.15.135:3300/0 2024-02-06T21:41:43.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff0e813f810 0x7ff0e813fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:43.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 10 monclient(hunting): picked mon.noname-a con 0x7ff0e813f810 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:43.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:43.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e8143510 con 0x7ff0e813f810 2024-02-06T21:41:43.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:43.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e80aa5a0 con 0x7ff0e8142bb0 2024-02-06T21:41:43.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:43.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e8142720 con 0x7ff0e80a9c80 2024-02-06T21:41:43.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:43.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.514+0000 7ff0ed80f640 10 monclient(hunting): authenticate will time out at 1333.852490s 2024-02-06T21:41:43.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.515+0000 7ff0e6ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e80a9c80 0x7ff0e80aa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:43.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.515+0000 7ff0e67fc640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff0e8142bb0 0x7ff0e8142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:43.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.515+0000 7ff0e6ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e80a9c80 0x7ff0e80aa060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:49226/0 (socket says 172.21.15.120:49226) 2024-02-06T21:41:43.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.515+0000 7ff0e6ffd640 1 -- 172.21.15.120:0/1134229907 learned_addr learned my addr 172.21.15.120:0/1134229907 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:43.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.515+0000 7ff0e6ffd640 10 monclient(hunting): get_auth_request con 0x7ff0e80a9c80 auth_method 0 2024-02-06T21:41:43.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.515+0000 7ff0e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:43.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.515+0000 7ff0e6ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:43.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.515+0000 7ff0e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:43.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.515+0000 7ff0e67fc640 10 monclient(hunting): get_auth_request con 0x7ff0e8142bb0 auth_method 0 2024-02-06T21:41:43.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.515+0000 7ff0e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:43.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.515+0000 7ff0e67fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:43.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.515+0000 7ff0e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:43.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.516+0000 7ff0e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:43.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.516+0000 7ff0e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:43.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.516+0000 7ff0e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:43.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.516+0000 7ff0e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:43.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.516+0000 7ff0e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:43.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.516+0000 7ff0e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:43.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.517+0000 7ff0e77fe640 1 --2- 172.21.15.120:0/1134229907 >> v2:172.21.15.120:3300/0 conn(0x7ff0e813f810 0x7ff0e813fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:43.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.517+0000 7ff0e77fe640 10 monclient(hunting): get_auth_request con 0x7ff0e813f810 auth_method 0 2024-02-06T21:41:43.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.517+0000 7ff0e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:43.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.517+0000 7ff0e77fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:43.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.517+0000 7ff0e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:43.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.518+0000 7ff0e67fc640 10 monclient(hunting): handle_auth_done global_id 4384 payload 274 2024-02-06T21:41:43.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.518+0000 7ff0e67fc640 1 -- 172.21.15.120:0/1134229907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e80a9c80 msgr2=0x7ff0e80aa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:43.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.518+0000 7ff0e67fc640 1 --2- 172.21.15.120:0/1134229907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e80a9c80 0x7ff0e80aa060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.518+0000 7ff0e67fc640 1 -- 172.21.15.120:0/1134229907 >> v2:172.21.15.120:3300/0 conn(0x7ff0e813f810 msgr2=0x7ff0e813fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:41:43.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.518+0000 7ff0e67fc640 1 --2- 172.21.15.120:0/1134229907 >> v2:172.21.15.120:3300/0 conn(0x7ff0e813f810 0x7ff0e813fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.518+0000 7ff0e67fc640 10 monclient: _finish_hunting 0 2024-02-06T21:41:43.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.518+0000 7ff0e67fc640 1 monclient: found mon.noname-c 2024-02-06T21:41:43.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.518+0000 7ff0e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:43.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.518+0000 7ff0e67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T21:41:43.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.518+0000 7ff0e67fc640 1 -- 172.21.15.120:0/1134229907 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e80ab120 con 0x7ff0e8142bb0 2024-02-06T21:41:43.524 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.520+0000 7ff0e67fc640 1 --2- 172.21.15.120:0/1134229907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0e8142bb0 0x7ff0e8142fd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff0e806cae0 tx=0x7ff0dc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1da86ce542694aea server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:43.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0e5ffb640 1 -- 172.21.15.120:0/1134229907 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc02e070 con 0x7ff0e8142bb0 2024-02-06T21:41:43.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:43.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0e5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T21:41:43.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0e5ffb640 10 monclient: dump: 2024-02-06T21:41:43.525 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:43.525 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:43.525 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:43.526 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:43.526 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:43.526 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:43.526 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:43.526 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:43.526 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:43.526 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:43.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0e5ffb640 10 monclient: _finish_auth 0 2024-02-06T21:41:43.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0e5ffb640 10 monclient: _check_auth_tickets 2024-02-06T21:41:43.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:43.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0e5ffb640 1 -- 172.21.15.120:0/1134229907 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0dc0315a0 con 0x7ff0e8142bb0 2024-02-06T21:41:43.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0ed80f640 5 monclient: authenticate success, global_id 4384 2024-02-06T21:41:43.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0ed80f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:41:43.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:43.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0ed80f640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:43.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0ed80f640 4 set_mon_vals no callback set 2024-02-06T21:41:43.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0e5ffb640 1 -- 172.21.15.120:0/1134229907 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc031930 con 0x7ff0e8142bb0 2024-02-06T21:41:43.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:43.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:43.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.523+0000 7ff0e5ffb640 10 monclient: dump: 2024-02-06T21:41:43.527 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:43.527 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:43.527 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:43.527 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:43.527 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:43.527 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:43.528 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:43.528 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:43.528 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:43.528 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:43.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.524+0000 7ff0ed80f640 10 monclient: shutdown 2024-02-06T21:41:43.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.524+0000 7ff0ed80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:43.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.524+0000 7ff0ed80f640 1 -- 172.21.15.120:0/1134229907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0e8142bb0 msgr2=0x7ff0e8142fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:43.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.524+0000 7ff0ed80f640 1 --2- 172.21.15.120:0/1134229907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0e8142bb0 0x7ff0e8142fd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff0e806cae0 tx=0x7ff0dc02b840 comp rx=0 tx=0).stop 2024-02-06T21:41:43.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.524+0000 7ff0ed80f640 1 -- 172.21.15.120:0/1134229907 shutdown_connections 2024-02-06T21:41:43.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.524+0000 7ff0ed80f640 1 --2- 172.21.15.120:0/1134229907 >> v2:172.21.15.120:3300/0 conn(0x7ff0e813f810 0x7ff0e813fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.524+0000 7ff0ed80f640 1 --2- 172.21.15.120:0/1134229907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0e8142bb0 0x7ff0e8142fd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.524+0000 7ff0ed80f640 1 --2- 172.21.15.120:0/1134229907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e80a9c80 0x7ff0e80aa060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.524+0000 7ff0ed80f640 1 -- 172.21.15.120:0/1134229907 >> 172.21.15.120:0/1134229907 conn(0x7ff0e80b4ba0 msgr2=0x7ff0e80b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:43.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.524+0000 7ff0ed80f640 1 -- 172.21.15.120:0/1134229907 shutdown_connections 2024-02-06T21:41:43.532 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.524+0000 7ff0ed80f640 1 -- 172.21.15.120:0/1134229907 wait complete. 2024-02-06T21:41:43.532 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.525+0000 7ff0ed80f640 10 monclient: build_initial_monmap 2024-02-06T21:41:43.532 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.525+0000 7ff0ed80f640 10 monclient: monmap: 2024-02-06T21:41:43.532 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:43.532 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:43.532 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:43.532 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:43.532 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:43.532 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:43.532 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:43.532 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:43.532 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:43.532 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:43.532 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.525+0000 7ff0ed80f640 1 librados: starting msgr at 2024-02-06T21:41:43.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.525+0000 7ff0ed80f640 1 librados: starting objecter 2024-02-06T21:41:43.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.525+0000 7ff0ed80f640 1 Processor -- start 2024-02-06T21:41:43.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.525+0000 7ff0ed80f640 1 -- start start 2024-02-06T21:41:43.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.525+0000 7ff0ed80f640 1 librados: setting wanted keys 2024-02-06T21:41:43.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.525+0000 7ff0ed80f640 1 librados: calling monclient init 2024-02-06T21:41:43.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.525+0000 7ff0ed80f640 10 monclient: init 2024-02-06T21:41:43.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:43.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T21:41:43.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0e80a9c80 0x7ff0e8141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:43.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 10 monclient(hunting): picked mon.noname-b con 0x7ff0e80a9c80 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:43.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e813f810 0x7ff0e80b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:43.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 10 monclient(hunting): picked mon.noname-c con 0x7ff0e813f810 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:43.533 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0e8142bb0 0x7ff0e81db680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:43.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 10 monclient(hunting): picked mon.noname-a con 0x7ff0e8142bb0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:43.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:43.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e81db1f0 con 0x7ff0e8142bb0 2024-02-06T21:41:43.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:43.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e80b75e0 con 0x7ff0e80a9c80 2024-02-06T21:41:43.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:43.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e80b61d0 con 0x7ff0e813f810 2024-02-06T21:41:43.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:43.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0ed80f640 10 monclient(hunting): authenticate will time out at 1333.863882s 2024-02-06T21:41:43.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0e67fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e813f810 0x7ff0e80b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:43.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.526+0000 7ff0e6ffd640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0e80a9c80 0x7ff0e8141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:43.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e67fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e813f810 0x7ff0e80b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:49242/0 (socket says 172.21.15.120:49242) 2024-02-06T21:41:43.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e67fc640 1 -- 172.21.15.120:0/1596660673 learned_addr learned my addr 172.21.15.120:0/1596660673 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:43.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e67fc640 10 monclient(hunting): get_auth_request con 0x7ff0e813f810 auth_method 0 2024-02-06T21:41:43.534 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e67fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e6ffd640 10 monclient(hunting): get_auth_request con 0x7ff0e80a9c80 auth_method 0 2024-02-06T21:41:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e6ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.527+0000 7ff0e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.528+0000 7ff0e77fe640 1 --2- 172.21.15.120:0/1596660673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0e8142bb0 0x7ff0e81db680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.529+0000 7ff0e77fe640 10 monclient(hunting): get_auth_request con 0x7ff0e8142bb0 auth_method 0 2024-02-06T21:41:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.529+0000 7ff0e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.529+0000 7ff0e77fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.529+0000 7ff0e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.530+0000 7ff0e6ffd640 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2024-02-06T21:41:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.530+0000 7ff0e6ffd640 1 -- 172.21.15.120:0/1596660673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e813f810 msgr2=0x7ff0e80b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.530+0000 7ff0e6ffd640 1 --2- 172.21.15.120:0/1596660673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e813f810 0x7ff0e80b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.530+0000 7ff0e6ffd640 1 -- 172.21.15.120:0/1596660673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0e8142bb0 msgr2=0x7ff0e81db680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.530+0000 7ff0e6ffd640 1 --2- 172.21.15.120:0/1596660673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0e8142bb0 0x7ff0e81db680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.530+0000 7ff0e6ffd640 10 monclient: _finish_hunting 0 2024-02-06T21:41:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.530+0000 7ff0e6ffd640 1 monclient: found mon.noname-b 2024-02-06T21:41:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.530+0000 7ff0e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.530+0000 7ff0e6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.530+0000 7ff0e6ffd640 1 -- 172.21.15.120:0/1596660673 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc009400 con 0x7ff0e80a9c80 2024-02-06T21:41:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.532+0000 7ff0e6ffd640 1 --2- 172.21.15.120:0/1596660673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0e80a9c80 0x7ff0e8141ee0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800e880 tx=0x7ff0d800ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.532+0000 7ff0cffff640 10 client.?.objecter ms_handle_connect 0x7ff0e80a9c80 2024-02-06T21:41:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.532+0000 7ff0cffff640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:43.539 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0cffff640 1 -- 172.21.15.120:0/1596660673 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d800c940 con 0x7ff0e80a9c80 2024-02-06T21:41:43.539 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:43.539 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0cffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:43.539 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0cffff640 10 monclient: dump: 2024-02-06T21:41:43.539 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:43.539 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:43.539 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:43.539 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:43.539 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:43.539 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:43.539 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:43.539 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:43.539 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:43.539 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:43.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0cffff640 10 monclient: _finish_auth 0 2024-02-06T21:41:43.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0cffff640 10 monclient: _check_auth_tickets 2024-02-06T21:41:43.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:43.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0cffff640 1 -- 172.21.15.120:0/1596660673 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d800caa0 con 0x7ff0e80a9c80 2024-02-06T21:41:43.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0cffff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:43.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0cffff640 1 -- 172.21.15.120:0/1596660673 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d800ce30 con 0x7ff0e80a9c80 2024-02-06T21:41:43.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0ed80f640 5 monclient: authenticate success, global_id 4387 2024-02-06T21:41:43.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0ed80f640 10 monclient: _renew_subs 2024-02-06T21:41:43.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0ed80f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:43.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0ed80f640 1 -- 172.21.15.120:0/1596660673 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e8145870 con 0x7ff0e80a9c80 2024-02-06T21:41:43.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0e57fa640 4 set_mon_vals no callback set 2024-02-06T21:41:43.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:43.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:43.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0cffff640 10 monclient: dump: 2024-02-06T21:41:43.541 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:43.541 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:43.541 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:43.541 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:43.541 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:43.541 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:43.541 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:43.541 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:43.541 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:43.541 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:43.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0ed80f640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:43.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0ed80f640 10 monclient: _renew_subs 2024-02-06T21:41:43.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0ed80f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:43.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0ed80f640 1 -- 172.21.15.120:0/1596660673 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e80b7b30 con 0x7ff0e80a9c80 2024-02-06T21:41:43.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.536+0000 7ff0ed80f640 1 librados: init done 2024-02-06T21:41:43.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.537+0000 7ff0ed80f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:43.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.537+0000 7ff0ed80f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:43.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.537+0000 7ff0ed80f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:43.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.537+0000 7ff0ed80f640 1 -- 172.21.15.120:0/1596660673 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e80686f0 con 0x7ff0e80a9c80 2024-02-06T21:41:43.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.784+0000 7ff0cffff640 1 -- 172.21.15.120:0/1596660673 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff0d8013470 con 0x7ff0e80a9c80 2024-02-06T21:41:43.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.785+0000 7ff0cffff640 1 --2- 172.21.15.120:0/1596660673 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff0d406d1a0 0x7ff0d406f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:43.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.785+0000 7ff0e67fc640 1 --2- 172.21.15.120:0/1596660673 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff0d406d1a0 0x7ff0d406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:43.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.785+0000 7ff0e67fc640 10 monclient: get_auth_request con 0x7ff0d406d1a0 auth_method 0 2024-02-06T21:41:43.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.786+0000 7ff0e67fc640 1 --2- 172.21.15.120:0/1596660673 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff0d406d1a0 0x7ff0d406f660 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc02b880 tx=0x7ff0dc03d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:43.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.786+0000 7ff0cffff640 10 client.4387.objecter ms_handle_connect 0x7ff0d406d1a0 2024-02-06T21:41:43.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.788+0000 7ff0cffff640 1 -- 172.21.15.120:0/1596660673 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff0d8059340 con 0x7ff0e80a9c80 2024-02-06T21:41:43.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.788+0000 7ff0cffff640 10 client.4387.objecter ms_dispatch 0x7ff0e8000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:43.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.788+0000 7ff0cffff640 3 client.4387.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:43.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.788+0000 7ff0cffff640 3 client.4387.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:43.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.788+0000 7ff0cffff640 20 client.4387.objecter dump_active .. 0 homeless 2024-02-06T21:41:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.800+0000 7ff0cffff640 1 -- 172.21.15.120:0/1596660673 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff0d80555e0 con 0x7ff0e80a9c80 2024-02-06T21:41:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.800+0000 7ff0cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.800+0000 7ff0cffff640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:43.976 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.975+0000 7ff0ed80f640 1 -- 172.21.15.120:0/1596660673 --> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff0e80ab670 con 0x7ff0d406d1a0 2024-02-06T21:41:43.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.977+0000 7ff0cffff640 1 -- 172.21.15.120:0/1596660673 <== mgr.4100 v2:172.21.15.135:6800/315664144 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47983 (secure 0 0 0) 0x7ff0dc031a80 con 0x7ff0d406d1a0 2024-02-06T21:41:43.979 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:41:43.979 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2024-02-06T21:41:43.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.979+0000 7ff0ed80f640 10 librados: watch_flush enter 2024-02-06T21:41:43.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.979+0000 7ff0ed80f640 10 librados: watch_flush exit 2024-02-06T21:41:43.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.979+0000 7ff0ed80f640 20 client.4387.objecter shutdown clearing up homeless session... 2024-02-06T21:41:43.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.979+0000 7ff0ed80f640 10 client.4387.objecter successfully canceled tick 2024-02-06T21:41:43.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.979+0000 7ff0ed80f640 1 -- 172.21.15.120:0/1596660673 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff0d406d1a0 msgr2=0x7ff0d406f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:43.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.979+0000 7ff0ed80f640 1 --2- 172.21.15.120:0/1596660673 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff0d406d1a0 0x7ff0d406f660 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc02b880 tx=0x7ff0dc03d040 comp rx=0 tx=0).stop 2024-02-06T21:41:43.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.979+0000 7ff0ed80f640 10 monclient: shutdown 2024-02-06T21:41:43.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.979+0000 7ff0ed80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:43.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.979+0000 7ff0ed80f640 1 -- 172.21.15.120:0/1596660673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0e80a9c80 msgr2=0x7ff0e8141ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:43.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.979+0000 7ff0ed80f640 1 --2- 172.21.15.120:0/1596660673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0e80a9c80 0x7ff0e8141ee0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800e880 tx=0x7ff0d800ed40 comp rx=0 tx=0).stop 2024-02-06T21:41:43.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.980+0000 7ff0ed80f640 1 -- 172.21.15.120:0/1596660673 shutdown_connections 2024-02-06T21:41:43.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.980+0000 7ff0ed80f640 1 --2- 172.21.15.120:0/1596660673 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff0d406d1a0 0x7ff0d406f660 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.980+0000 7ff0ed80f640 1 --2- 172.21.15.120:0/1596660673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0e8142bb0 0x7ff0e81db680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.980+0000 7ff0ed80f640 1 --2- 172.21.15.120:0/1596660673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e813f810 0x7ff0e80b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.980+0000 7ff0ed80f640 1 --2- 172.21.15.120:0/1596660673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0e80a9c80 0x7ff0e8141ee0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:43.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.980+0000 7ff0ed80f640 1 -- 172.21.15.120:0/1596660673 >> 172.21.15.120:0/1596660673 conn(0x7ff0e80b4ba0 msgr2=0x7ff0e813a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:43.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.980+0000 7ff0ed80f640 1 -- 172.21.15.120:0/1596660673 shutdown_connections 2024-02-06T21:41:43.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.980+0000 7ff0ed80f640 1 -- 172.21.15.120:0/1596660673 wait complete. 2024-02-06T21:41:43.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:43.980+0000 7ff0ed80f640 1 librados: shutdown 2024-02-06T21:41:43.992 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-02-06T21:41:43.569748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219064,"kb_used_data":2544,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755656,"statfs":{"total":773094113280,"available":772869791744,"internally_reserved":0,"allocated":2605056,"data_stored":1409904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.201796"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:36.722429+0000","last_change":"2024-02-06T21:41:36.144443+0000","last_active":"2024-02-06T21:41:36.722429+0000","last_peered":"2024-02-06T21:41:36.722429+0000","last_clean":"2024-02-06T21:41:36.722429+0000","last_became_active":"2024-02-06T21:41:36.139312+0000","last_became_peered":"2024-02-06T21:41:36.139312+0000","last_unstale":"2024-02-06T21:41:36.722429+0000","last_undegraded":"2024-02-06T21:41:36.722429+0000","last_fullsized":"2024-02-06T21:41:36.722429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:47:25.348858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:36.641182+0000","last_change":"2024-02-06T21:41:36.193098+0000","last_active":"2024-02-06T21:41:36.641182+0000","last_peered":"2024-02-06T21:41:36.641182+0000","last_clean":"2024-02-06T21:41:36.641182+0000","last_became_active":"2024-02-06T21:41:36.160648+0000","last_became_peered":"2024-02-06T21:41:36.160648+0000","last_unstale":"2024-02-06T21:41:36.641182+0000","last_undegraded":"2024-02-06T21:41:36.641182+0000","last_fullsized":"2024-02-06T21:41:36.641182+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T09:01:03.161738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:36.725737+0000","last_change":"2024-02-06T21:41:35.766413+0000","last_active":"2024-02-06T21:41:36.725737+0000","last_peered":"2024-02-06T21:41:36.725737+0000","last_clean":"2024-02-06T21:41:36.725737+0000","last_became_active":"2024-02-06T21:41:35.761324+0000","last_became_peered":"2024-02-06T21:41:35.761324+0000","last_unstale":"2024-02-06T21:41:36.725737+0000","last_undegraded":"2024-02-06T21:41:36.725737+0000","last_fullsized":"2024-02-06T21:41:36.725737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:28:07.018583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:36.518526+0000","last_change":"2024-02-06T21:41:35.918723+0000","last_active":"2024-02-06T21:41:36.518526+0000","last_peered":"2024-02-06T21:41:36.518526+0000","last_clean":"2024-02-06T21:41:36.518526+0000","last_became_active":"2024-02-06T21:41:35.891748+0000","last_became_peered":"2024-02-06T21:41:35.891748+0000","last_unstale":"2024-02-06T21:41:36.518526+0000","last_undegraded":"2024-02-06T21:41:36.518526+0000","last_fullsized":"2024-02-06T21:41:36.518526+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T03:59:11.081636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:36.719554+0000","last_change":"2024-02-06T21:41:35.892547+0000","last_active":"2024-02-06T21:41:36.719554+0000","last_peered":"2024-02-06T21:41:36.719554+0000","last_clean":"2024-02-06T21:41:36.719554+0000","last_became_active":"2024-02-06T21:41:35.877657+0000","last_became_peered":"2024-02-06T21:41:35.877657+0000","last_unstale":"2024-02-06T21:41:36.719554+0000","last_undegraded":"2024-02-06T21:41:36.719554+0000","last_fullsized":"2024-02-06T21:41:36.719554+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:23:28.851375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:37.171759+0000","last_change":"2024-02-06T21:41:35.636555+0000","last_active":"2024-02-06T21:41:37.171759+0000","last_peered":"2024-02-06T21:41:37.171759+0000","last_clean":"2024-02-06T21:41:37.171759+0000","last_became_active":"2024-02-06T21:41:35.603135+0000","last_became_peered":"2024-02-06T21:41:35.603135+0000","last_unstale":"2024-02-06T21:41:37.171759+0000","last_undegraded":"2024-02-06T21:41:37.171759+0000","last_fullsized":"2024-02-06T21:41:37.171759+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T03:13:14.699865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:36.708204+0000","last_change":"2024-02-06T21:41:35.688242+0000","last_active":"2024-02-06T21:41:36.708204+0000","last_peered":"2024-02-06T21:41:36.708204+0000","last_clean":"2024-02-06T21:41:36.708204+0000","last_became_active":"2024-02-06T21:41:35.642164+0000","last_became_peered":"2024-02-06T21:41:35.642164+0000","last_unstale":"2024-02-06T21:41:36.708204+0000","last_undegraded":"2024-02-06T21:41:36.708204+0000","last_fullsized":"2024-02-06T21:41:36.708204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:50:03.838525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:36.715056+0000","last_change":"2024-02-06T21:41:36.168152+0000","last_active":"2024-02-06T21:41:36.715056+0000","last_peered":"2024-02-06T21:41:36.715056+0000","last_clean":"2024-02-06T21:41:36.715056+0000","last_became_active":"2024-02-06T21:41:36.162988+0000","last_became_peered":"2024-02-06T21:41:36.162988+0000","last_unstale":"2024-02-06T21:41:36.715056+0000","last_undegraded":"2024-02-06T21:41:36.715056+0000","last_fullsized":"2024-02-06T21:41:36.715056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T00:47:41.526670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'64","reported_seq":106,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:37.153955+0000","last_change":"2024-02-06T21:41:31.339576+0000","last_active":"2024-02-06T21:41:37.153955+0000","last_peered":"2024-02-06T21:41:37.153955+0000","last_clean":"2024-02-06T21:41:37.153955+0000","last_became_active":"2024-02-06T21:41:31.305766+0000","last_became_peered":"2024-02-06T21:41:31.305766+0000","last_unstale":"2024-02-06T21:41:37.153955+0000","last_undegraded":"2024-02-06T21:41:37.153955+0000","last_fullsized":"2024-02-06T21:41:37.153955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:20.409569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:20.409569+0000","last_clean_scrub_stamp":"2024-02-06T21:41:20.409569+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:24:57.767580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":18,"seq":77309411337,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":626688,"data_stored":474618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.576000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.736000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.289999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.705}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.281000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.172999999999998}]}]},{"osd":6,"up_from":17,"seq":73014444041,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":76778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.5839999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.745999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.808}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.205}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.196000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.336}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.244}]}]},{"osd":5,"up_from":17,"seq":73014444043,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":626688,"data_stored":474618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.0330000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4979999999999993}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.853}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.927}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.468999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.438000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.664000000000001}]}]},{"osd":4,"up_from":17,"seq":73014444041,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":76778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.692}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.377000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.672000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.655999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0060000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.029}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.812000000000001}]}]},{"osd":3,"up_from":17,"seq":73014444041,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":76778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.143000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.407999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.427}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.603000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1400000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.356999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.393000000000001}]}]},{"osd":2,"up_from":17,"seq":73014444041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":76778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.056999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.041}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.177}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":165.429}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.113}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8460000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3010000000000002}]}]},{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":76778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.259}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.066000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.9400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":43.347000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.804}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.676}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":114.285}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":76778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.428000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.039999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.077999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8040000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4510000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.053000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.035}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-06T21:41:43.994 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-06T21:41:44.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.093+0000 7f2902cb3640 10 monclient: get_monmap_and_config 2024-02-06T21:41:44.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.093+0000 7f2902cb3640 10 monclient: build_initial_monmap 2024-02-06T21:41:44.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.093+0000 7f2902cb3640 10 monclient: monmap: 2024-02-06T21:41:44.096 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:44.096 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:44.096 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:44.095201+0000 2024-02-06T21:41:44.096 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:44.095201+0000 2024-02-06T21:41:44.096 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:44.096 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:44.096 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:44.096 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:44.096 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:44.096 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:44.096 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:44.096 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:44.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.094+0000 7f2902cb3640 1 Processor -- start 2024-02-06T21:41:44.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.094+0000 7f2902cb3640 1 -- start start 2024-02-06T21:41:44.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.094+0000 7f2902cb3640 10 monclient: init 2024-02-06T21:41:44.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.095+0000 7f2902cb3640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:44.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.095+0000 7f2902cb3640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-02-06T21:41:44.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.095+0000 7f2902cb3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28fc0b54b0 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:44.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.095+0000 7f2902cb3640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f28fc13ff40 0x7f28fc0a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:44.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.095+0000 7f2902cb3640 10 monclient(hunting): picked mon.noname-a con 0x7f28fc13ff40 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:44.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.095+0000 7f2902cb3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f28fc142680 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:44.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.095+0000 7f2902cb3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc143200 con 0x7f28fc142680 2024-02-06T21:41:44.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.095+0000 7f2902cb3640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc0ab720 con 0x7f28fc0b54b0 2024-02-06T21:41:44.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.095+0000 7f2902cb3640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:44.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.095+0000 7f2902cb3640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28fc0ab520 con 0x7f28fc13ff40 2024-02-06T21:41:44.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.095+0000 7f2902cb3640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:44.099 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.095+0000 7f2902cb3640 10 monclient(hunting): authenticate will time out at 1334.432895s 2024-02-06T21:41:44.099 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.096+0000 7f2900a4f640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f28fc0b54b0 0x7f28fc13f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.120:53598/0 (socket says 172.21.15.120:53598) 2024-02-06T21:41:44.099 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.096+0000 7f2900a4f640 1 -- 172.21.15.120:0/121740211 learned_addr learned my addr 172.21.15.120:0/121740211 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:41:44.102 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.101+0000 7f28fbfff640 1 --2- 172.21.15.120:0/121740211 >> v2:172.21.15.120:3300/0 conn(0x7f28fc13ff40 0x7f28fc0a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:44.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.101+0000 7f28fbfff640 10 monclient(hunting): get_auth_request con 0x7f28fc13ff40 auth_method 0 2024-02-06T21:41:44.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.101+0000 7f28fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:44.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.101+0000 7f28fbfff640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:44.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.101+0000 7f28fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:44.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.102+0000 7f28fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:44.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.102+0000 7f28fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:44.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.102+0000 7f28fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:44.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.105+0000 7f28fbfff640 10 monclient(hunting): handle_auth_done global_id 4389 payload 274 2024-02-06T21:41:44.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.105+0000 7f28fbfff640 1 -- 172.21.15.120:0/121740211 >> v1:172.21.15.135:6789/0 conn(0x7f28fc0b54b0 legacy=0x7f28fc13f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:44.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.105+0000 7f28fbfff640 1 -- 172.21.15.120:0/121740211 >> v1:172.21.15.120:6789/0 conn(0x7f28fc142680 legacy=0x7f28fc142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:44.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.105+0000 7f28fbfff640 10 monclient: _finish_hunting 0 2024-02-06T21:41:44.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.105+0000 7f28fbfff640 1 monclient: found mon.noname-a 2024-02-06T21:41:44.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.105+0000 7f28fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:44.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.105+0000 7f28fbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:44.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.105+0000 7f28fbfff640 1 -- 172.21.15.120:0/121740211 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc0b4800 con 0x7f28fc13ff40 2024-02-06T21:41:44.109 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.108+0000 7f28fbfff640 1 --2- 172.21.15.120:0/121740211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28fc13ff40 0x7f28fc0a9860 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f28e80097a0 tx=0x7f28e802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a145176778f52b62 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:44.118 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f28fb7fe640 1 -- 172.21.15.120:0/121740211 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8030610 con 0x7f28fc13ff40 2024-02-06T21:41:44.118 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f28fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:44.118 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f28fb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:44.118 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f28fb7fe640 10 monclient: dump: 2024-02-06T21:41:44.118 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:44.118 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:44.118 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.118 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.118 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:44.118 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:44.118 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:44.118 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:44.118 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:44.118 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:44.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f28fb7fe640 10 monclient: _finish_auth 0 2024-02-06T21:41:44.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f28fb7fe640 10 monclient: _check_auth_tickets 2024-02-06T21:41:44.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f28fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:44.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f28fb7fe640 1 -- 172.21.15.120:0/121740211 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e8030770 con 0x7f28fc13ff40 2024-02-06T21:41:44.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f28fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:44.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f28fb7fe640 1 -- 172.21.15.120:0/121740211 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8030b00 con 0x7f28fc13ff40 2024-02-06T21:41:44.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f28fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:44.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f28fb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:44.119 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f28fb7fe640 10 monclient: dump: 2024-02-06T21:41:44.119 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:44.119 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:44.119 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.119 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.119 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:44.120 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:44.120 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:44.120 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:44.120 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:44.120 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:44.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f2902cb3640 5 monclient: authenticate success, global_id 4389 2024-02-06T21:41:44.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f2902cb3640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:44.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.115+0000 7f2902cb3640 4 set_mon_vals no callback set 2024-02-06T21:41:44.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.116+0000 7f2902cb3640 10 monclient: shutdown 2024-02-06T21:41:44.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.116+0000 7f2902cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:44.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.116+0000 7f2902cb3640 1 -- 172.21.15.120:0/121740211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28fc13ff40 msgr2=0x7f28fc0a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:44.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.116+0000 7f2902cb3640 1 --2- 172.21.15.120:0/121740211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28fc13ff40 0x7f28fc0a9860 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f28e80097a0 tx=0x7f28e802b960 comp rx=0 tx=0).stop 2024-02-06T21:41:44.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.116+0000 7f2902cb3640 1 -- 172.21.15.120:0/121740211 shutdown_connections 2024-02-06T21:41:44.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.116+0000 7f2902cb3640 1 --2- 172.21.15.120:0/121740211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28fc13ff40 0x7f28fc0a9860 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:44.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.116+0000 7f2902cb3640 1 -- 172.21.15.120:0/121740211 >> 172.21.15.120:0/121740211 conn(0x7f28fc0b8660 msgr2=0x7f28fc0b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:44.121 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.116+0000 7f2902cb3640 1 -- 172.21.15.120:0/121740211 shutdown_connections 2024-02-06T21:41:44.121 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.116+0000 7f2902cb3640 1 -- 172.21.15.120:0/121740211 wait complete. 2024-02-06T21:41:44.122 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.117+0000 7f2902cb3640 10 monclient: build_initial_monmap 2024-02-06T21:41:44.122 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.117+0000 7f2902cb3640 10 monclient: monmap: 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.117+0000 7f2902cb3640 1 librados: starting msgr at 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.117+0000 7f2902cb3640 1 librados: starting objecter 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.117+0000 7f2902cb3640 1 Processor -- start 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.117+0000 7f2902cb3640 1 -- start start 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.117+0000 7f2902cb3640 1 librados: setting wanted keys 2024-02-06T21:41:44.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.117+0000 7f2902cb3640 1 librados: calling monclient init 2024-02-06T21:41:44.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.117+0000 7f2902cb3640 10 monclient: init 2024-02-06T21:41:44.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:44.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 10 monclient: _add_conns ranks=[0,1,2] 2024-02-06T21:41:44.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28fc0b54b0 0x7f28fc0b6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:44.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 10 monclient(hunting): picked mon.noname-a con 0x7f28fc0b54b0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:44.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f28fc13ff40 0x7f28fc137060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:44.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 10 monclient(hunting): picked mon.noname-b con 0x7f28fc13ff40 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:44.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28fc142680 0x7f28fc1375a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:44.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 10 monclient(hunting): picked mon.noname-c con 0x7f28fc142680 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:44.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:44.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc136880 con 0x7f28fc0b54b0 2024-02-06T21:41:44.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:44.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc1385b0 con 0x7f28fc13ff40 2024-02-06T21:41:44.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:44.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc138890 con 0x7f28fc142680 2024-02-06T21:41:44.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:44.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2902cb3640 10 monclient(hunting): authenticate will time out at 1334.455782s 2024-02-06T21:41:44.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.118+0000 7f2900a4f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28fc0b54b0 0x7f28fc0b6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:44.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.119+0000 7f2900a4f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28fc0b54b0 0x7f28fc0b6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:33378/0 (socket says 172.21.15.120:33378) 2024-02-06T21:41:44.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.119+0000 7f2900a4f640 1 -- 172.21.15.120:0/2928474788 learned_addr learned my addr 172.21.15.120:0/2928474788 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:44.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.119+0000 7f2900a4f640 10 monclient(hunting): get_auth_request con 0x7f28fc0b54b0 auth_method 0 2024-02-06T21:41:44.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.119+0000 7f2900a4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:44.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.119+0000 7f2900a4f640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:44.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.119+0000 7f2900a4f640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:44.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.119+0000 7f2900a4f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:44.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.119+0000 7f2900a4f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:44.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.119+0000 7f2900a4f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:44.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.122+0000 7f2900a4f640 10 monclient(hunting): handle_auth_done global_id 4395 payload 824 2024-02-06T21:41:44.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.122+0000 7f2900a4f640 1 -- 172.21.15.120:0/2928474788 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28fc142680 msgr2=0x7f28fc1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:44.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.122+0000 7f2900a4f640 1 --2- 172.21.15.120:0/2928474788 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28fc142680 0x7f28fc1375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:44.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.122+0000 7f2900a4f640 1 -- 172.21.15.120:0/2928474788 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f28fc13ff40 msgr2=0x7f28fc137060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:44.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.122+0000 7f2900a4f640 1 --2- 172.21.15.120:0/2928474788 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f28fc13ff40 0x7f28fc137060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:44.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.122+0000 7f2900a4f640 10 monclient: _finish_hunting 0 2024-02-06T21:41:44.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.122+0000 7f2900a4f640 1 monclient: found mon.noname-a 2024-02-06T21:41:44.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.122+0000 7f2900a4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:44.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.122+0000 7f2900a4f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:44.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.122+0000 7f2900a4f640 1 -- 172.21.15.120:0/2928474788 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e8009400 con 0x7f28fc0b54b0 2024-02-06T21:41:44.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.124+0000 7f2900a4f640 1 --2- 172.21.15.120:0/2928474788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28fc0b54b0 0x7f28fc0b6c70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f28f000c480 tx=0x7f28f000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:44.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.124+0000 7f28f97fa640 10 client.?.objecter ms_handle_connect 0x7f28fc0b54b0 2024-02-06T21:41:44.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.124+0000 7f28f97fa640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:44.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f28f97fa640 1 -- 172.21.15.120:0/2928474788 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0005ce0 con 0x7f28fc0b54b0 2024-02-06T21:41:44.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f28f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:44.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f28f97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:44.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f28f97fa640 10 monclient: dump: 2024-02-06T21:41:44.129 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:44.129 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:44.129 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.129 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.129 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:44.129 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:44.129 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:44.130 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:44.130 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:44.130 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:44.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f28f97fa640 10 monclient: _finish_auth 0 2024-02-06T21:41:44.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f28f97fa640 10 monclient: _check_auth_tickets 2024-02-06T21:41:44.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f28f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:44.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f2902cb3640 5 monclient: authenticate success, global_id 4395 2024-02-06T21:41:44.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f2902cb3640 10 monclient: _renew_subs 2024-02-06T21:41:44.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f2902cb3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:44.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f2902cb3640 1 -- 172.21.15.120:0/2928474788 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28fc138b70 con 0x7f28fc0b54b0 2024-02-06T21:41:44.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f2902cb3640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:44.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f2902cb3640 10 monclient: _renew_subs 2024-02-06T21:41:44.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f2902cb3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:44.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f2902cb3640 1 -- 172.21.15.120:0/2928474788 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28fc138e50 con 0x7f28fc0b54b0 2024-02-06T21:41:44.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.127+0000 7f2902cb3640 1 librados: init done 2024-02-06T21:41:44.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.128+0000 7f2902cb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:44.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.128+0000 7f2902cb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:44.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.128+0000 7f2902cb3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T21:41:44.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.128+0000 7f2902cb3640 1 -- 172.21.15.120:0/2928474788 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28fc04ef90 con 0x7f28fc0b54b0 2024-02-06T21:41:44.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.130+0000 7f28f97fa640 1 -- 172.21.15.120:0/2928474788 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f0001140 con 0x7f28fc0b54b0 2024-02-06T21:41:44.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.130+0000 7f28f97fa640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:44.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.130+0000 7f28f97fa640 1 -- 172.21.15.120:0/2928474788 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f00072f0 con 0x7f28fc0b54b0 2024-02-06T21:41:44.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.130+0000 7f28f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:44.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.130+0000 7f28f97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:44.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.130+0000 7f28f97fa640 10 monclient: dump: 2024-02-06T21:41:44.132 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:44.132 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:44.132 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.132 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.132 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:44.132 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:44.132 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:44.132 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:44.133 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:44.133 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:44.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.130+0000 7f28faffd640 4 set_mon_vals no callback set 2024-02-06T21:41:44.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.401+0000 7f28f97fa640 1 -- 172.21.15.120:0/2928474788 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f28f0007510 con 0x7f28fc0b54b0 2024-02-06T21:41:44.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.402+0000 7f28f97fa640 1 --2- 172.21.15.120:0/2928474788 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f28e006d1a0 0x7f28e006f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:44.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.402+0000 7f28fbfff640 1 --2- 172.21.15.120:0/2928474788 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f28e006d1a0 0x7f28e006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:44.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.402+0000 7f28fbfff640 10 monclient: get_auth_request con 0x7f28e006d1a0 auth_method 0 2024-02-06T21:41:44.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.403+0000 7f28fbfff640 1 --2- 172.21.15.120:0/2928474788 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f28e006d1a0 0x7f28e006f660 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f28e8036000 tx=0x7f28e803b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:44.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.403+0000 7f28f97fa640 10 client.4395.objecter ms_handle_connect 0x7f28e006d1a0 2024-02-06T21:41:44.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.421+0000 7f28f97fa640 1 -- 172.21.15.120:0/2928474788 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f28f0058760 con 0x7f28fc0b54b0 2024-02-06T21:41:44.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.421+0000 7f28f97fa640 10 client.4395.objecter ms_dispatch 0x7f28fc000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:44.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.421+0000 7f28f97fa640 3 client.4395.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:44.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.421+0000 7f28f97fa640 3 client.4395.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:44.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.421+0000 7f28f97fa640 20 client.4395.objecter dump_active .. 0 homeless 2024-02-06T21:41:44.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.421+0000 7f28f97fa640 1 -- 172.21.15.120:0/2928474788 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f28f0054a00 con 0x7f28fc0b54b0 2024-02-06T21:41:44.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.421+0000 7f28f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:44.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.421+0000 7f28f97fa640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:44.598 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.597+0000 7f2902cb3640 1 -- 172.21.15.120:0/2928474788 --> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f28fc0aa450 con 0x7f28e006d1a0 2024-02-06T21:41:44.601 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.599+0000 7f28f97fa640 1 -- 172.21.15.120:0/2928474788 <== mgr.4100 v2:172.21.15.135:6800/315664144 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47983 (secure 0 0 0) 0x7f28e8030960 con 0x7f28e006d1a0 2024-02-06T21:41:44.601 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:41:44.602 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2024-02-06T21:41:44.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 10 librados: watch_flush enter 2024-02-06T21:41:44.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 10 librados: watch_flush exit 2024-02-06T21:41:44.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 20 client.4395.objecter shutdown clearing up homeless session... 2024-02-06T21:41:44.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 10 client.4395.objecter successfully canceled tick 2024-02-06T21:41:44.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 1 -- 172.21.15.120:0/2928474788 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f28e006d1a0 msgr2=0x7f28e006f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:44.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 1 --2- 172.21.15.120:0/2928474788 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f28e006d1a0 0x7f28e006f660 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f28e8036000 tx=0x7f28e803b040 comp rx=0 tx=0).stop 2024-02-06T21:41:44.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 10 monclient: shutdown 2024-02-06T21:41:44.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:44.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 1 -- 172.21.15.120:0/2928474788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28fc0b54b0 msgr2=0x7f28fc0b6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:44.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 1 --2- 172.21.15.120:0/2928474788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28fc0b54b0 0x7f28fc0b6c70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f28f000c480 tx=0x7f28f000c940 comp rx=0 tx=0).stop 2024-02-06T21:41:44.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 1 -- 172.21.15.120:0/2928474788 shutdown_connections 2024-02-06T21:41:44.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 1 --2- 172.21.15.120:0/2928474788 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f28e006d1a0 0x7f28e006f660 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:44.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 1 --2- 172.21.15.120:0/2928474788 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f28fc13ff40 0x7f28fc137060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:44.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 1 --2- 172.21.15.120:0/2928474788 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28fc142680 0x7f28fc1375a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:44.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 1 --2- 172.21.15.120:0/2928474788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28fc0b54b0 0x7f28fc0b6c70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:44.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 1 -- 172.21.15.120:0/2928474788 >> 172.21.15.120:0/2928474788 conn(0x7f28fc0b8660 msgr2=0x7f28fc142ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:44.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 1 -- 172.21.15.120:0/2928474788 shutdown_connections 2024-02-06T21:41:44.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 1 -- 172.21.15.120:0/2928474788 wait complete. 2024-02-06T21:41:44.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.601+0000 7f2902cb3640 1 librados: shutdown 2024-02-06T21:41:44.614 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-02-06T21:41:43.569748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219064,"kb_used_data":2544,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755656,"statfs":{"total":773094113280,"available":772869791744,"internally_reserved":0,"allocated":2605056,"data_stored":1409904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.201796"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:36.722429+0000","last_change":"2024-02-06T21:41:36.144443+0000","last_active":"2024-02-06T21:41:36.722429+0000","last_peered":"2024-02-06T21:41:36.722429+0000","last_clean":"2024-02-06T21:41:36.722429+0000","last_became_active":"2024-02-06T21:41:36.139312+0000","last_became_peered":"2024-02-06T21:41:36.139312+0000","last_unstale":"2024-02-06T21:41:36.722429+0000","last_undegraded":"2024-02-06T21:41:36.722429+0000","last_fullsized":"2024-02-06T21:41:36.722429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:47:25.348858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:36.641182+0000","last_change":"2024-02-06T21:41:36.193098+0000","last_active":"2024-02-06T21:41:36.641182+0000","last_peered":"2024-02-06T21:41:36.641182+0000","last_clean":"2024-02-06T21:41:36.641182+0000","last_became_active":"2024-02-06T21:41:36.160648+0000","last_became_peered":"2024-02-06T21:41:36.160648+0000","last_unstale":"2024-02-06T21:41:36.641182+0000","last_undegraded":"2024-02-06T21:41:36.641182+0000","last_fullsized":"2024-02-06T21:41:36.641182+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T09:01:03.161738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:36.725737+0000","last_change":"2024-02-06T21:41:35.766413+0000","last_active":"2024-02-06T21:41:36.725737+0000","last_peered":"2024-02-06T21:41:36.725737+0000","last_clean":"2024-02-06T21:41:36.725737+0000","last_became_active":"2024-02-06T21:41:35.761324+0000","last_became_peered":"2024-02-06T21:41:35.761324+0000","last_unstale":"2024-02-06T21:41:36.725737+0000","last_undegraded":"2024-02-06T21:41:36.725737+0000","last_fullsized":"2024-02-06T21:41:36.725737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:28:07.018583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:36.518526+0000","last_change":"2024-02-06T21:41:35.918723+0000","last_active":"2024-02-06T21:41:36.518526+0000","last_peered":"2024-02-06T21:41:36.518526+0000","last_clean":"2024-02-06T21:41:36.518526+0000","last_became_active":"2024-02-06T21:41:35.891748+0000","last_became_peered":"2024-02-06T21:41:35.891748+0000","last_unstale":"2024-02-06T21:41:36.518526+0000","last_undegraded":"2024-02-06T21:41:36.518526+0000","last_fullsized":"2024-02-06T21:41:36.518526+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T03:59:11.081636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:36.719554+0000","last_change":"2024-02-06T21:41:35.892547+0000","last_active":"2024-02-06T21:41:36.719554+0000","last_peered":"2024-02-06T21:41:36.719554+0000","last_clean":"2024-02-06T21:41:36.719554+0000","last_became_active":"2024-02-06T21:41:35.877657+0000","last_became_peered":"2024-02-06T21:41:35.877657+0000","last_unstale":"2024-02-06T21:41:36.719554+0000","last_undegraded":"2024-02-06T21:41:36.719554+0000","last_fullsized":"2024-02-06T21:41:36.719554+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:23:28.851375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:37.171759+0000","last_change":"2024-02-06T21:41:35.636555+0000","last_active":"2024-02-06T21:41:37.171759+0000","last_peered":"2024-02-06T21:41:37.171759+0000","last_clean":"2024-02-06T21:41:37.171759+0000","last_became_active":"2024-02-06T21:41:35.603135+0000","last_became_peered":"2024-02-06T21:41:35.603135+0000","last_unstale":"2024-02-06T21:41:37.171759+0000","last_undegraded":"2024-02-06T21:41:37.171759+0000","last_fullsized":"2024-02-06T21:41:37.171759+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T03:13:14.699865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:36.708204+0000","last_change":"2024-02-06T21:41:35.688242+0000","last_active":"2024-02-06T21:41:36.708204+0000","last_peered":"2024-02-06T21:41:36.708204+0000","last_clean":"2024-02-06T21:41:36.708204+0000","last_became_active":"2024-02-06T21:41:35.642164+0000","last_became_peered":"2024-02-06T21:41:35.642164+0000","last_unstale":"2024-02-06T21:41:36.708204+0000","last_undegraded":"2024-02-06T21:41:36.708204+0000","last_fullsized":"2024-02-06T21:41:36.708204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:50:03.838525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:36.715056+0000","last_change":"2024-02-06T21:41:36.168152+0000","last_active":"2024-02-06T21:41:36.715056+0000","last_peered":"2024-02-06T21:41:36.715056+0000","last_clean":"2024-02-06T21:41:36.715056+0000","last_became_active":"2024-02-06T21:41:36.162988+0000","last_became_peered":"2024-02-06T21:41:36.162988+0000","last_unstale":"2024-02-06T21:41:36.715056+0000","last_undegraded":"2024-02-06T21:41:36.715056+0000","last_fullsized":"2024-02-06T21:41:36.715056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T00:47:41.526670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'64","reported_seq":106,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-02-06T21:41:37.153955+0000","last_change":"2024-02-06T21:41:31.339576+0000","last_active":"2024-02-06T21:41:37.153955+0000","last_peered":"2024-02-06T21:41:37.153955+0000","last_clean":"2024-02-06T21:41:37.153955+0000","last_became_active":"2024-02-06T21:41:31.305766+0000","last_became_peered":"2024-02-06T21:41:31.305766+0000","last_unstale":"2024-02-06T21:41:37.153955+0000","last_undegraded":"2024-02-06T21:41:37.153955+0000","last_fullsized":"2024-02-06T21:41:37.153955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:20.409569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:20.409569+0000","last_clean_scrub_stamp":"2024-02-06T21:41:20.409569+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:24:57.767580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":18,"seq":77309411337,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":626688,"data_stored":474618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.576000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.736000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.289999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.705}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.281000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.172999999999998}]}]},{"osd":6,"up_from":17,"seq":73014444041,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":76778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.5839999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.745999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.808}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.205}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.196000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.336}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.244}]}]},{"osd":5,"up_from":17,"seq":73014444043,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":626688,"data_stored":474618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.0330000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4979999999999993}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.853}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.927}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.468999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.438000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.664000000000001}]}]},{"osd":4,"up_from":17,"seq":73014444041,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":76778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.692}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.377000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.672000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.655999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0060000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.029}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.812000000000001}]}]},{"osd":3,"up_from":17,"seq":73014444041,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":76778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.143000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.407999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.427}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.603000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1400000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.356999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.393000000000001}]}]},{"osd":2,"up_from":17,"seq":73014444041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":76778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.056999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.041}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.177}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":165.429}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.113}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8460000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3010000000000002}]}]},{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":76778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.259}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.066000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.9400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":43.347000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.804}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.676}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":114.285}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":76778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.428000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.039999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.077999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8040000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4510000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.053000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.035}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-06T21:41:44.615 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-02-06T21:41:44.615 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-06T21:41:44.615 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-02-06T21:41:44.615 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-06T21:41:44.715 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.713+0000 7f23cd633640 10 monclient: get_monmap_and_config 2024-02-06T21:41:44.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.713+0000 7f23cd633640 10 monclient: build_initial_monmap 2024-02-06T21:41:44.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.714+0000 7f23cd633640 10 monclient: monmap: 2024-02-06T21:41:44.716 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:44.716 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:44.716 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:41:44.715268+0000 2024-02-06T21:41:44.716 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:41:44.715268+0000 2024-02-06T21:41:44.716 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:44.716 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:44.716 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T21:41:44.716 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T21:41:44.716 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T21:41:44.716 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T21:41:44.717 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T21:41:44.717 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:44.717 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.714+0000 7f23cd633640 1 Processor -- start 2024-02-06T21:41:44.717 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.714+0000 7f23cd633640 1 -- start start 2024-02-06T21:41:44.717 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.714+0000 7f23cd633640 10 monclient: init 2024-02-06T21:41:44.718 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.715+0000 7f23cd633640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:44.718 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.715+0000 7f23cd633640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-02-06T21:41:44.718 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.715+0000 7f23cd633640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23c80a9c80 addr v1:172.21.15.135:6789/0 2024-02-06T21:41:44.718 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.715+0000 7f23cd633640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23c80aa060 addr v1:172.21.15.120:6789/0 2024-02-06T21:41:44.718 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.715+0000 7f23cd633640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f23c813ff20 0x7f23c814de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:44.719 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.715+0000 7f23cd633640 10 monclient(hunting): picked mon.noname-a con 0x7f23c813ff20 addr v2:172.21.15.120:3300/0 2024-02-06T21:41:44.719 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.715+0000 7f23cd633640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c8142720 con 0x7f23c80aa060 2024-02-06T21:41:44.719 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.715+0000 7f23cd633640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c80ab120 con 0x7f23c80a9c80 2024-02-06T21:41:44.719 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.715+0000 7f23cd633640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:44.719 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.715+0000 7f23cd633640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c80ab390 con 0x7f23c813ff20 2024-02-06T21:41:44.719 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.715+0000 7f23cd633640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:44.719 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.715+0000 7f23cd633640 10 monclient(hunting): authenticate will time out at 1335.052749s 2024-02-06T21:41:44.719 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.715+0000 7f23c77fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f23c813ff20 0x7f23c814de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:44.719 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.716+0000 7f23c6ffd640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f23c80a9c80 0x7f23c8142bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.120:53606/0 (socket says 172.21.15.120:53606) 2024-02-06T21:41:44.719 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.716+0000 7f23c6ffd640 1 -- 172.21.15.120:0/952400791 learned_addr learned my addr 172.21.15.120:0/952400791 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T21:41:44.719 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.716+0000 7f23c77fe640 10 monclient(hunting): get_auth_request con 0x7f23c813ff20 auth_method 0 2024-02-06T21:41:44.719 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.716+0000 7f23c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:44.719 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.716+0000 7f23c77fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:44.719 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.716+0000 7f23c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:44.719 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.716+0000 7f23c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:44.720 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.716+0000 7f23c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:44.720 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.717+0000 7f23c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:44.720 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.718+0000 7f23c5ffb640 1 -- 172.21.15.120:0/952400791 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430816613 0 0) 0x7f23c80ab120 con 0x7f23c80a9c80 2024-02-06T21:41:44.720 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.718+0000 7f23c5ffb640 10 monclient(hunting): _negotiate 2024-02-06T21:41:44.721 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.718+0000 7f23c5ffb640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:44.721 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.718+0000 7f23c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:44.721 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.718+0000 7f23c5ffb640 10 monclient(hunting): my global_id is 4393 2024-02-06T21:41:44.721 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.718+0000 7f23c5ffb640 1 -- 172.21.15.120:0/952400791 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b0003520 con 0x7f23c80a9c80 2024-02-06T21:41:44.721 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.719+0000 7f23c77fe640 10 monclient(hunting): handle_auth_done global_id 4398 payload 274 2024-02-06T21:41:44.721 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.719+0000 7f23c77fe640 1 -- 172.21.15.120:0/952400791 >> v1:172.21.15.135:6789/0 conn(0x7f23c80a9c80 legacy=0x7f23c8142bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:44.721 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.719+0000 7f23c77fe640 1 -- 172.21.15.120:0/952400791 >> v1:172.21.15.120:6789/0 conn(0x7f23c80aa060 legacy=0x7f23c813f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:44.721 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.719+0000 7f23c77fe640 10 monclient: _finish_hunting 0 2024-02-06T21:41:44.721 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.719+0000 7f23c77fe640 1 monclient: found mon.noname-a 2024-02-06T21:41:44.721 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.719+0000 7f23c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:44.721 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.719+0000 7f23c77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T21:41:44.721 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.719+0000 7f23c77fe640 1 -- 172.21.15.120:0/952400791 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c80ab670 con 0x7f23c813ff20 2024-02-06T21:41:44.722 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.721+0000 7f23c77fe640 1 --2- 172.21.15.120:0/952400791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23c813ff20 0x7f23c814de90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f23c00097a0 tx=0x7f23c002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc6a5e8b6534722 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:44.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23c5ffb640 1 -- 172.21.15.120:0/952400791 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c002f810 con 0x7f23c813ff20 2024-02-06T21:41:44.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:44.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23c5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T21:41:44.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23c5ffb640 10 monclient: dump: 2024-02-06T21:41:44.732 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:44.732 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:44.732 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.732 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.732 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:44.732 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:44.732 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:44.732 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:44.732 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:44.732 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:44.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23c5ffb640 10 monclient: _finish_auth 0 2024-02-06T21:41:44.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23c5ffb640 10 monclient: _check_auth_tickets 2024-02-06T21:41:44.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:44.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23c5ffb640 1 -- 172.21.15.120:0/952400791 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c002f970 con 0x7f23c813ff20 2024-02-06T21:41:44.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:44.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23c5ffb640 1 -- 172.21.15.120:0/952400791 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c002fd00 con 0x7f23c813ff20 2024-02-06T21:41:44.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23cd633640 5 monclient: authenticate success, global_id 4398 2024-02-06T21:41:44.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23cd633640 10 monclient: get_monmap_and_config success 2024-02-06T21:41:44.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23cd633640 4 set_mon_vals no callback set 2024-02-06T21:41:44.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:44.734 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T21:41:44.734 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23c5ffb640 10 monclient: dump: 2024-02-06T21:41:44.734 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:44.734 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:44.734 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.734 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.734 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:44.734 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:44.734 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:44.734 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:44.735 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:44.735 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:44.740 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23cd633640 10 monclient: shutdown 2024-02-06T21:41:44.740 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23cd633640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:44.740 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23cd633640 1 -- 172.21.15.120:0/952400791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23c813ff20 msgr2=0x7f23c814de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:44.740 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.729+0000 7f23cd633640 1 --2- 172.21.15.120:0/952400791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23c813ff20 0x7f23c814de90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f23c00097a0 tx=0x7f23c002b9f0 comp rx=0 tx=0).stop 2024-02-06T21:41:44.740 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.730+0000 7f23cd633640 1 -- 172.21.15.120:0/952400791 shutdown_connections 2024-02-06T21:41:44.740 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.730+0000 7f23cd633640 1 --2- 172.21.15.120:0/952400791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23c813ff20 0x7f23c814de90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:44.740 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.730+0000 7f23cd633640 1 -- 172.21.15.120:0/952400791 >> 172.21.15.120:0/952400791 conn(0x7f23c80b4ba0 msgr2=0x7f23c80b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:44.740 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.730+0000 7f23cd633640 1 -- 172.21.15.120:0/952400791 shutdown_connections 2024-02-06T21:41:44.741 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.730+0000 7f23cd633640 1 -- 172.21.15.120:0/952400791 wait complete. 2024-02-06T21:41:44.741 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.730+0000 7f23cd633640 10 monclient: build_initial_monmap 2024-02-06T21:41:44.741 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.730+0000 7f23cd633640 10 monclient: monmap: 2024-02-06T21:41:44.741 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T21:41:44.741 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:41:44.741 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T21:41:44.741 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T21:41:44.741 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T21:41:44.741 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:44.741 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:41:44.741 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:41:44.742 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:41:44.742 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:44.742 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.730+0000 7f23cd633640 1 librados: starting msgr at 2024-02-06T21:41:44.742 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.730+0000 7f23cd633640 1 librados: starting objecter 2024-02-06T21:41:44.742 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 1 Processor -- start 2024-02-06T21:41:44.742 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 1 -- start start 2024-02-06T21:41:44.742 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 1 librados: setting wanted keys 2024-02-06T21:41:44.742 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 1 librados: calling monclient init 2024-02-06T21:41:44.742 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 10 monclient: init 2024-02-06T21:41:44.742 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 10 monclient: _reopen_session rank -1 2024-02-06T21:41:44.742 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T21:41:44.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f23c80a9c80 0x7f23c8144420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:44.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 10 monclient(hunting): picked mon.noname-b con 0x7f23c80a9c80 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:41:44.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23c80aa060 0x7f23c80b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:44.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 10 monclient(hunting): picked mon.noname-a con 0x7f23c80aa060 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:41:44.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23c813ff20 0x7f23c80b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:44.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 10 monclient(hunting): picked mon.noname-c con 0x7f23c813ff20 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:41:44.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:44.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80b1b20 con 0x7f23c80aa060 2024-02-06T21:41:44.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:44.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c8146530 con 0x7f23c80a9c80 2024-02-06T21:41:44.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 10 monclient(hunting): start opening mon connection 2024-02-06T21:41:44.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c8146810 con 0x7f23c813ff20 2024-02-06T21:41:44.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 10 monclient(hunting): _renew_subs 2024-02-06T21:41:44.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.731+0000 7f23cd633640 10 monclient(hunting): authenticate will time out at 1335.069406s 2024-02-06T21:41:44.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c77fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23c813ff20 0x7f23c80b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:44.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c67fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23c80aa060 0x7f23c80b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:44.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c6ffd640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f23c80a9c80 0x7f23c8144420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:44.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c77fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23c813ff20 0x7f23c80b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:49248/0 (socket says 172.21.15.120:49248) 2024-02-06T21:41:44.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c67fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23c80aa060 0x7f23c80b10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:33390/0 (socket says 172.21.15.120:33390) 2024-02-06T21:41:44.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c77fe640 1 -- 172.21.15.120:0/4265560257 learned_addr learned my addr 172.21.15.120:0/4265560257 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:41:44.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c77fe640 10 monclient(hunting): get_auth_request con 0x7f23c813ff20 auth_method 0 2024-02-06T21:41:44.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:44.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c77fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:44.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:44.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c67fc640 10 monclient(hunting): get_auth_request con 0x7f23c80aa060 auth_method 0 2024-02-06T21:41:44.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:44.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c67fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:44.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:44.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c6ffd640 10 monclient(hunting): get_auth_request con 0x7f23c80a9c80 auth_method 0 2024-02-06T21:41:44.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:41:44.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c6ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:41:44.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.732+0000 7f23c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:41:44.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.733+0000 7f23c67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:44.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.733+0000 7f23c67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:44.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.733+0000 7f23c67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:44.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.733+0000 7f23c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:44.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.733+0000 7f23c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:44.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.733+0000 7f23c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:44.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.733+0000 7f23c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:41:44.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.733+0000 7f23c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:41:44.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.733+0000 7f23c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:41:44.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.736+0000 7f23c6ffd640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2024-02-06T21:41:44.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.736+0000 7f23c6ffd640 1 -- 172.21.15.120:0/4265560257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23c813ff20 msgr2=0x7f23c80b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:44.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.736+0000 7f23c6ffd640 1 --2- 172.21.15.120:0/4265560257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23c813ff20 0x7f23c80b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:44.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.736+0000 7f23c6ffd640 1 -- 172.21.15.120:0/4265560257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23c80aa060 msgr2=0x7f23c80b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:44.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.736+0000 7f23c6ffd640 1 --2- 172.21.15.120:0/4265560257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23c80aa060 0x7f23c80b10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:44.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.736+0000 7f23c6ffd640 10 monclient: _finish_hunting 0 2024-02-06T21:41:44.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.736+0000 7f23c6ffd640 1 monclient: found mon.noname-b 2024-02-06T21:41:44.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.736+0000 7f23c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:41:44.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.736+0000 7f23c6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:41:44.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.736+0000 7f23c6ffd640 1 -- 172.21.15.120:0/4265560257 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c0009400 con 0x7f23c80a9c80 2024-02-06T21:41:44.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.736+0000 7f23c67fc640 1 --2- 172.21.15.120:0/4265560257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23c80aa060 0x7f23c80b10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-06T21:41:44.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.736+0000 7f23c77fe640 1 --2- 172.21.15.120:0/4265560257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23c813ff20 0x7f23c80b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-06T21:41:44.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.738+0000 7f23c6ffd640 1 --2- 172.21.15.120:0/4265560257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f23c80a9c80 0x7f23c8144420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f23b800cbc0 tx=0x7f23b800a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:44.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.738+0000 7f23af7fe640 10 client.?.objecter ms_handle_connect 0x7f23c80a9c80 2024-02-06T21:41:44.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.738+0000 7f23af7fe640 10 client.?.objecter resend_mon_ops 2024-02-06T21:41:44.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.741+0000 7f23af7fe640 1 -- 172.21.15.120:0/4265560257 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b800aca0 con 0x7f23c80a9c80 2024-02-06T21:41:44.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.741+0000 7f23af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:44.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.741+0000 7f23af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:41:44.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.741+0000 7f23af7fe640 10 monclient: dump: 2024-02-06T21:41:44.749 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:44.749 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:44.750 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.750 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.750 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:44.750 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:44.750 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:44.750 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:44.750 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:44.750 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:44.750 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.741+0000 7f23af7fe640 10 monclient: _finish_auth 0 2024-02-06T21:41:44.750 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.741+0000 7f23af7fe640 10 monclient: _check_auth_tickets 2024-02-06T21:41:44.750 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.741+0000 7f23af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:41:44.750 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.741+0000 7f23af7fe640 1 -- 172.21.15.120:0/4265560257 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b800ae00 con 0x7f23c80a9c80 2024-02-06T21:41:44.750 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.741+0000 7f23af7fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:41:44.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.741+0000 7f23af7fe640 1 -- 172.21.15.120:0/4265560257 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8012600 con 0x7f23c80a9c80 2024-02-06T21:41:44.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23cd633640 5 monclient: authenticate success, global_id 4396 2024-02-06T21:41:44.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23cd633640 10 monclient: _renew_subs 2024-02-06T21:41:44.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23cd633640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:44.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23cd633640 1 -- 172.21.15.120:0/4265560257 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c8145640 con 0x7f23c80a9c80 2024-02-06T21:41:44.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:41:44.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:41:44.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23af7fe640 10 monclient: dump: 2024-02-06T21:41:44.751 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T21:41:44.751 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:41:44.751 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.751 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:41:44.751 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T21:41:44.752 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T21:41:44.752 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:41:44.752 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:41:44.752 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:41:44.752 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T21:41:44.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23c57fa640 4 set_mon_vals no callback set 2024-02-06T21:41:44.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23cd633640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:41:44.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23cd633640 10 monclient: _renew_subs 2024-02-06T21:41:44.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23cd633640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:44.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23cd633640 1 -- 172.21.15.120:0/4265560257 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c8145190 con 0x7f23c80a9c80 2024-02-06T21:41:44.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23cd633640 1 librados: init done 2024-02-06T21:41:44.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23cd633640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:44.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23cd633640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:44.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23cd633640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:44.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.742+0000 7f23cd633640 1 -- 172.21.15.120:0/4265560257 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c804ef90 con 0x7f23c80a9c80 2024-02-06T21:41:44.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.761+0000 7f23af7fe640 1 -- 172.21.15.120:0/4265560257 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f23b8007070 con 0x7f23c80a9c80 2024-02-06T21:41:44.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.761+0000 7f23af7fe640 1 --2- 172.21.15.120:0/4265560257 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f23b406d1f0 0x7f23b406f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:41:44.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.761+0000 7f23c67fc640 1 --2- 172.21.15.120:0/4265560257 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f23b406d1f0 0x7f23b406f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:41:44.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.761+0000 7f23c67fc640 10 monclient: get_auth_request con 0x7f23b406d1f0 auth_method 0 2024-02-06T21:41:44.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.762+0000 7f23af7fe640 10 client.4396.objecter ms_handle_connect 0x7f23b406d1f0 2024-02-06T21:41:44.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.762+0000 7f23c67fc640 1 --2- 172.21.15.120:0/4265560257 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f23b406d1f0 0x7f23b406f6b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f23c81dbe70 tx=0x7f23bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:41:44.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.989+0000 7f23af7fe640 1 -- 172.21.15.120:0/4265560257 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f23b8058df0 con 0x7f23c80a9c80 2024-02-06T21:41:44.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.989+0000 7f23af7fe640 10 client.4396.objecter ms_dispatch 0x7f23c8000b90 osd_map(23..23 src has 1..23) v4 2024-02-06T21:41:44.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.989+0000 7f23af7fe640 3 client.4396.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:41:44.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.989+0000 7f23af7fe640 3 client.4396.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:41:44.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.989+0000 7f23af7fe640 20 client.4396.objecter dump_active .. 0 homeless 2024-02-06T21:41:45.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.999+0000 7f23af7fe640 1 -- 172.21.15.120:0/4265560257 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f23b8055090 con 0x7f23c80a9c80 2024-02-06T21:41:45.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.999+0000 7f23af7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T21:41:45.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:44.999+0000 7f23af7fe640 10 monclient: _finish_command 1 = system:0 2024-02-06T21:41:45.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.213+0000 7f23cd633640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-02-06T21:41:45.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.213+0000 7f23cd633640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-02-06T21:41:45.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.213+0000 7f23cd633640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:41:45.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.213+0000 7f23cd633640 1 -- 172.21.15.120:0/4265560257 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f23c8069e30 con 0x7f23c80a9c80 2024-02-06T21:41:45.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.221+0000 7f23af7fe640 1 -- 172.21.15.120:0/4265560257 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f23c8069e30 con 0x7f23c80a9c80 2024-02-06T21:41:45.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.221+0000 7f23af7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-02-06T21:41:45.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.221+0000 7f23af7fe640 10 monclient: _finish_command 2 = system:0 2024-02-06T21:41:45.222 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T21:41:45.222 INFO:teuthology.orchestra.run.smithi120.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-06T21:41:45.224 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 10 librados: watch_flush enter 2024-02-06T21:41:45.224 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 10 librados: watch_flush exit 2024-02-06T21:41:45.224 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 20 client.4396.objecter shutdown clearing up homeless session... 2024-02-06T21:41:45.224 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 10 client.4396.objecter successfully canceled tick 2024-02-06T21:41:45.224 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 1 -- 172.21.15.120:0/4265560257 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f23b406d1f0 msgr2=0x7f23b406f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:45.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 1 --2- 172.21.15.120:0/4265560257 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f23b406d1f0 0x7f23b406f6b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f23c81dbe70 tx=0x7f23bc009040 comp rx=0 tx=0).stop 2024-02-06T21:41:45.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 10 monclient: shutdown 2024-02-06T21:41:45.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:41:45.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 1 -- 172.21.15.120:0/4265560257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f23c80a9c80 msgr2=0x7f23c8144420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:41:45.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 1 --2- 172.21.15.120:0/4265560257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f23c80a9c80 0x7f23c8144420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f23b800cbc0 tx=0x7f23b800a060 comp rx=0 tx=0).stop 2024-02-06T21:41:45.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 1 -- 172.21.15.120:0/4265560257 shutdown_connections 2024-02-06T21:41:45.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 1 --2- 172.21.15.120:0/4265560257 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f23b406d1f0 0x7f23b406f6b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:45.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 1 --2- 172.21.15.120:0/4265560257 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23c813ff20 0x7f23c80b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:45.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 1 --2- 172.21.15.120:0/4265560257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23c80aa060 0x7f23c80b10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:45.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 1 --2- 172.21.15.120:0/4265560257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f23c80a9c80 0x7f23c8144420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:41:45.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 1 -- 172.21.15.120:0/4265560257 >> 172.21.15.120:0/4265560257 conn(0x7f23c80b4ba0 msgr2=0x7f23c813a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:41:45.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 1 -- 172.21.15.120:0/4265560257 shutdown_connections 2024-02-06T21:41:45.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 1 -- 172.21.15.120:0/4265560257 wait complete. 2024-02-06T21:41:45.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T21:41:45.222+0000 7f23cd633640 1 librados: shutdown 2024-02-06T21:41:45.235 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-02-06T21:41:45.235 INFO:teuthology.run_tasks:Running task workunit... 2024-02-06T21:41:45.249 INFO:tasks.workunit:Pulling workunits from ref 2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc 2024-02-06T21:41:45.249 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-06T21:41:45.249 DEBUG:teuthology.orchestra.run.smithi120:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-06T21:41:45.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-06T21:41:45.264 INFO:teuthology.orchestra.run.smithi120.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-02-06T21:41:45.264 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-02-06T21:41:45.319 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-02-06T21:41:45.319 DEBUG:teuthology.orchestra.run.smithi120:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-02-06T21:41:45.374 INFO:tasks.workunit:timeout=6h 2024-02-06T21:41:45.374 INFO:tasks.workunit:cleanup=True 2024-02-06T21:41:45.374 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ljflores/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc 2024-02-06T21:41:45.430 INFO:tasks.workunit.client.0.smithi120.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-06T21:41:49.464 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:01:29.278 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:49.469 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:01:29.284 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:49.511 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:01:29.326 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:49.518 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:01:29.332 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:49.518 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:01:29.281 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:49.519 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:01:29.283 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:49.529 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:01:29.343 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:49.566 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:01:29.380 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:41:49.567 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:01:29.331 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:09.472 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:01:49.286 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:09.554 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:01:49.369 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:09.571 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:01:49.386 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:09.606 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:01:49.370 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:09.607 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:01:49.372 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:09.621 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:01:49.385 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:09.627 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:01:49.442 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:09.632 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:01:49.447 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:09.657 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:01:49.472 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:29.531 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:02:09.346 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:29.580 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:02:09.394 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:29.625 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:02:09.440 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:29.634 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:09.398 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:29.639 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:09.404 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:29.643 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:02:09.457 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:29.648 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:02:09.463 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:29.655 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:09.418 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:29.669 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:02:09.484 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:34.478 INFO:tasks.workunit.client.0.smithi120.stderr:Note: switching to '2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc'. 2024-02-06T21:42:34.478 INFO:tasks.workunit.client.0.smithi120.stderr: 2024-02-06T21:42:34.478 INFO:tasks.workunit.client.0.smithi120.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-06T21:42:34.478 INFO:tasks.workunit.client.0.smithi120.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-06T21:42:34.478 INFO:tasks.workunit.client.0.smithi120.stderr:state without impacting any branches by switching back to a branch. 2024-02-06T21:42:34.478 INFO:tasks.workunit.client.0.smithi120.stderr: 2024-02-06T21:42:34.479 INFO:tasks.workunit.client.0.smithi120.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-06T21:42:34.479 INFO:tasks.workunit.client.0.smithi120.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-06T21:42:34.479 INFO:tasks.workunit.client.0.smithi120.stderr: 2024-02-06T21:42:34.479 INFO:tasks.workunit.client.0.smithi120.stderr: git switch -c 2024-02-06T21:42:34.479 INFO:tasks.workunit.client.0.smithi120.stderr: 2024-02-06T21:42:34.479 INFO:tasks.workunit.client.0.smithi120.stderr:Or undo this operation with: 2024-02-06T21:42:34.479 INFO:tasks.workunit.client.0.smithi120.stderr: 2024-02-06T21:42:34.479 INFO:tasks.workunit.client.0.smithi120.stderr: git switch - 2024-02-06T21:42:34.479 INFO:tasks.workunit.client.0.smithi120.stderr: 2024-02-06T21:42:34.479 INFO:tasks.workunit.client.0.smithi120.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-06T21:42:34.479 INFO:tasks.workunit.client.0.smithi120.stderr: 2024-02-06T21:42:34.479 INFO:tasks.workunit.client.0.smithi120.stderr:HEAD is now at 2b35b7e08ec Revert "qa/tasks/cephadm: enable mon_cluster_log_to_file" 2024-02-06T21:42:34.481 DEBUG:teuthology.orchestra.run.smithi120:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-06T21:42:34.542 INFO:tasks.workunit.client.0.smithi120.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-06T21:42:34.544 INFO:tasks.workunit.client.0.smithi120.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-06T21:42:34.544 INFO:tasks.workunit.client.0.smithi120.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-06T21:42:35.209 INFO:tasks.workunit.client.0.smithi120.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-06T21:42:35.281 INFO:tasks.workunit.client.0.smithi120.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-06T21:42:35.322 INFO:tasks.workunit.client.0.smithi120.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-06T21:42:35.323 INFO:tasks.workunit.client.0.smithi120.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-06T21:42:35.323 INFO:tasks.workunit.client.0.smithi120.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-06T21:42:35.362 INFO:tasks.workunit.client.0.smithi120.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-06T21:42:35.365 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-02-06T21:42:35.365 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-06T21:42:35.421 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-02-06T21:42:35.422 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-02-06T21:42:35.422 DEBUG:teuthology.orchestra.run.smithi120:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-02-06T21:42:35.481 INFO:tasks.workunit.client.0.smithi120.stderr:+ parallel=1 2024-02-06T21:42:35.482 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' '' = --serial ']' 2024-02-06T21:42:35.482 INFO:tasks.workunit.client.0.smithi120.stderr:+ crimson=0 2024-02-06T21:42:35.482 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' '' = --crimson ']' 2024-02-06T21:42:35.482 INFO:tasks.workunit.client.0.smithi120.stderr:+ color= 2024-02-06T21:42:35.482 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -t 1 ']' 2024-02-06T21:42:35.482 INFO:tasks.workunit.client.0.smithi120.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-02-06T21:42:35.482 INFO:tasks.workunit.client.0.smithi120.stderr:+ GTEST_OUTPUT=xml:/home/ubuntu/cephtest/archive/gtest_xml_report 2024-02-06T21:42:35.482 INFO:tasks.workunit.client.0.smithi120.stderr:+ declare -A pids 2024-02-06T21:42:35.482 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.482 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.482 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_aio 2024-02-06T21:42:35.482 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_aio' 2024-02-06T21:42:35.482 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_aio 2024-02-06T21:42:35.482 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.483 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_aio 2024-02-06T21:42:35.483 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40401 2024-02-06T21:42:35.483 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_aio on pid 40401' 2024-02-06T21:42:35.483 INFO:tasks.workunit.client.0.smithi120.stdout:test api_aio on pid 40401 2024-02-06T21:42:35.486 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40401 2024-02-06T21:42:35.486 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.486 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.487 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-02-06T21:42:35.487 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_aio_pp 2024-02-06T21:42:35.487 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_aio_pp' 2024-02-06T21:42:35.487 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_aio_pp 2024-02-06T21:42:35.487 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.487 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.487 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.487 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.487 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.487 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.487 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.487 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.487 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.487 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.488 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.488 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.488 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.488 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.488 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.488 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.488 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.488 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.488 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.488 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.488 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.488 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.488 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.488 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.489 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.489 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.489 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_aio_pp 2024-02-06T21:42:35.489 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40408 2024-02-06T21:42:35.489 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_aio_pp on pid 40408' 2024-02-06T21:42:35.489 INFO:tasks.workunit.client.0.smithi120.stdout:test api_aio_pp on pid 40408 2024-02-06T21:42:35.492 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40408 2024-02-06T21:42:35.492 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.492 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.492 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-02-06T21:42:35.493 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_io 2024-02-06T21:42:35.493 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_io' 2024-02-06T21:42:35.493 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_io 2024-02-06T21:42:35.493 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.493 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.493 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.493 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.493 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.493 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.493 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.493 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.493 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.493 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.493 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.493 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.494 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.494 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.494 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.494 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.494 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.494 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.494 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.494 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.494 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.494 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.494 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.494 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.494 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.494 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.494 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.495 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.495 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.495 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.495 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.495 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.495 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.495 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.495 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.495 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.495 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.495 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.495 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.495 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.495 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.495 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.496 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.496 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.496 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.496 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.496 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_io 2024-02-06T21:42:35.496 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40417 2024-02-06T21:42:35.496 INFO:tasks.workunit.client.0.smithi120.stdout:test api_io on pid 40417 2024-02-06T21:42:35.500 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_io on pid 40417' 2024-02-06T21:42:35.500 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40417 2024-02-06T21:42:35.500 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.500 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.500 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-02-06T21:42:35.500 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_io_pp 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_io_pp' 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_io_pp 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.501 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.502 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.503 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.503 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.503 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.503 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.503 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.503 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.503 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.503 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.503 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.503 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.503 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.503 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.503 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.503 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.503 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.504 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.504 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.504 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.504 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.504 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.504 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.504 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.504 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.504 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.504 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.504 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_io_pp 2024-02-06T21:42:35.504 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40428 2024-02-06T21:42:35.504 INFO:tasks.workunit.client.0.smithi120.stdout:test api_io_pp on pid 40428 2024-02-06T21:42:35.510 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_io_pp on pid 40428' 2024-02-06T21:42:35.510 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40428 2024-02-06T21:42:35.510 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.510 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-02-06T21:42:35.510 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.510 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_asio 2024-02-06T21:42:35.510 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_asio' 2024-02-06T21:42:35.510 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_asio 2024-02-06T21:42:35.510 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.510 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.510 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.510 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.510 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.511 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.511 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.511 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.511 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.511 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.511 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.511 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.511 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.511 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.511 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.511 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.511 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.511 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.511 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.511 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.512 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.512 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.512 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.512 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.512 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.512 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.512 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.512 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.512 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.512 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.512 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.512 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.512 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.512 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.512 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.513 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.514 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.515 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.515 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.515 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.515 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.515 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.515 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.515 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.515 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.515 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_asio 2024-02-06T21:42:35.515 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.515 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.515 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.515 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40442 2024-02-06T21:42:35.516 INFO:tasks.workunit.client.0.smithi120.stdout:test api_asio on pid 40442 2024-02-06T21:42:35.521 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_asio on pid 40442' 2024-02-06T21:42:35.521 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40442 2024-02-06T21:42:35.521 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.521 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-02-06T21:42:35.521 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.521 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_list 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_list' 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_list 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.522 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.528 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.528 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.528 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.528 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40401/exe 2024-02-06T21:42:35.528 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.528 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.528 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.528 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.528 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.528 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_list 2024-02-06T21:42:35.528 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40462 2024-02-06T21:42:35.528 INFO:tasks.workunit.client.0.smithi120.stdout:test api_list on pid 40462 2024-02-06T21:42:35.535 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_list on pid 40462' 2024-02-06T21:42:35.535 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40462 2024-02-06T21:42:35.535 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.535 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.535 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-02-06T21:42:35.536 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.536 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_lock 2024-02-06T21:42:35.536 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.536 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_lock' 2024-02-06T21:42:35.536 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.536 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.536 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_lock 2024-02-06T21:42:35.536 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.536 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.536 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.536 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.536 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.536 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.536 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.536 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.537 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.538 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.539 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.540 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.540 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.540 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.540 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.540 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.540 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.540 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.540 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.540 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.540 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.540 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.540 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.540 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_aio.log 2024-02-06T21:42:35.541 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_aio: /' 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40408/exe 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.542 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_lock 2024-02-06T21:42:35.543 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.543 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.543 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40479 2024-02-06T21:42:35.543 INFO:tasks.workunit.client.0.smithi120.stdout:test api_lock on pid 40479 2024-02-06T21:42:35.552 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_lock on pid 40479' 2024-02-06T21:42:35.552 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40479 2024-02-06T21:42:35.552 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.552 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.552 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-02-06T21:42:35.553 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.553 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_lock_pp 2024-02-06T21:42:35.553 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.553 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.553 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.553 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.553 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.553 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_lock_pp' 2024-02-06T21:42:35.553 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.553 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.553 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.553 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.553 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.553 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.553 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.554 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.554 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.554 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.554 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.554 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.554 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.554 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.554 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.554 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.554 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.554 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.554 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.554 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.554 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.554 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.555 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.555 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.555 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.555 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.555 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.555 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.555 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.555 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.555 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.555 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.555 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.555 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.555 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.555 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.555 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2024-02-06T21:42:35.556 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.557 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.558 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.559 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.559 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.559 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.559 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.559 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.559 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.559 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.559 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.559 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.559 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.559 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.559 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.559 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_aio_pp: /' 2024-02-06T21:42:35.559 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.559 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.560 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.560 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.560 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.560 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.560 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.560 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.560 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.560 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.560 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.560 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.560 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_lock_pp 2024-02-06T21:42:35.560 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.560 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.560 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.560 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_lock_pp 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40417/exe 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.561 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40502 2024-02-06T21:42:35.562 INFO:tasks.workunit.client.0.smithi120.stdout:test api_lock_pp on pid 40502 2024-02-06T21:42:35.573 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_lock_pp on pid 40502' 2024-02-06T21:42:35.573 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40502 2024-02-06T21:42:35.573 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.573 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.573 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.573 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.573 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-02-06T21:42:35.573 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.573 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.573 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_misc 2024-02-06T21:42:35.573 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_misc' 2024-02-06T21:42:35.574 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.574 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.574 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.574 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.574 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.574 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.574 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.574 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.574 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.574 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.574 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.574 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.574 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.574 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.575 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.576 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.576 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.576 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.576 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.576 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.576 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.576 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.576 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.576 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.576 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.576 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.576 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.576 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.576 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.576 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_misc 2024-02-06T21:42:35.577 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.578 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.579 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.580 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.580 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.580 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.580 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.580 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.580 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.580 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.580 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.580 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.580 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.580 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.580 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.580 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.580 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.580 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.581 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40428/exe 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_io.log 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.582 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.583 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.583 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_misc 2024-02-06T21:42:35.583 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.583 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.583 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40526 2024-02-06T21:42:35.583 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.583 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.583 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.583 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.583 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_misc on pid 40526' 2024-02-06T21:42:35.583 INFO:tasks.workunit.client.0.smithi120.stdout:test api_misc on pid 40526 2024-02-06T21:42:35.590 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40526 2024-02-06T21:42:35.590 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.590 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.590 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_misc_pp 2024-02-06T21:42:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_misc_pp' 2024-02-06T21:42:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-02-06T21:42:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.592 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.592 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.592 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.592 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.592 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.592 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.592 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.592 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.592 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.592 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_io: /' 2024-02-06T21:42:35.592 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.592 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.592 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.592 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.593 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.593 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.593 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.593 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.593 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.593 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.593 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.593 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.593 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.593 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.593 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.593 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.593 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.593 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.593 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.594 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.595 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.595 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.595 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.595 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.595 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.595 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.595 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-02-06T21:42:35.595 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.595 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_misc_pp 2024-02-06T21:42:35.595 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.595 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.595 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.595 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.595 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.595 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.596 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40442/exe 2024-02-06T21:42:35.596 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.596 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.596 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.596 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.596 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.596 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.596 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.596 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_io_pp: /' 2024-02-06T21:42:35.596 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_misc_pp 2024-02-06T21:42:35.596 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40545 2024-02-06T21:42:35.596 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.596 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.597 INFO:tasks.workunit.client.0.smithi120.stdout:test api_misc_pp on pid 40545 2024-02-06T21:42:35.602 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.602 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_misc_pp on pid 40545' 2024-02-06T21:42:35.602 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40545 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_tier_pp 2024-02-06T21:42:35.603 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_tier_pp' 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.604 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.605 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.605 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.605 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_asio: /' 2024-02-06T21:42:35.605 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.605 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.605 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.605 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.605 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.605 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.605 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40462/exe 2024-02-06T21:42:35.605 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.605 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.605 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.605 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.606 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.606 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.606 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.606 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.606 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.606 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.606 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.606 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.606 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.606 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.606 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-02-06T21:42:35.606 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2024-02-06T21:42:35.606 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.606 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2024-02-06T21:42:35.606 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_tier_pp 2024-02-06T21:42:35.607 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.607 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.607 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.607 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.607 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.607 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.607 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.607 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.607 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.607 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_tier_pp 2024-02-06T21:42:35.607 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.607 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.607 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.607 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.607 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40558 2024-02-06T21:42:35.608 INFO:tasks.workunit.client.0.smithi120.stdout:test api_tier_pp on pid 40558 2024-02-06T21:42:35.618 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_tier_pp on pid 40558' 2024-02-06T21:42:35.619 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.619 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40558 2024-02-06T21:42:35.619 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.619 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.619 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.619 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.619 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.619 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.619 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.619 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.619 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.619 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.619 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.620 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.620 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40479/exe 2024-02-06T21:42:35.620 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.620 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-02-06T21:42:35.620 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.620 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.620 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.620 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.620 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.620 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.620 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.620 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.620 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_asio.log 2024-02-06T21:42:35.620 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_list: /' 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.622 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2024-02-06T21:42:35.623 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.623 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.623 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.623 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.623 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.623 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.623 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.623 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.623 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.623 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.623 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.623 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.623 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.623 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.623 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.624 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.624 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.624 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.624 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.624 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_lock.log 2024-02-06T21:42:35.624 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.624 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.624 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.624 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.624 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.624 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.624 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.624 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.624 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.624 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_list.log 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.628 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.628 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.628 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.628 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.628 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.628 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.628 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.628 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.628 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.628 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.628 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_pool 2024-02-06T21:42:35.628 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.628 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.628 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.628 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.629 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.629 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.629 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_pool' 2024-02-06T21:42:35.629 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.629 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_pool 2024-02-06T21:42:35.629 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_pool 2024-02-06T21:42:35.629 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.629 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40582 2024-02-06T21:42:35.629 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_pool on pid 40582' 2024-02-06T21:42:35.629 INFO:tasks.workunit.client.0.smithi120.stdout:test api_pool on pid 40582 2024-02-06T21:42:35.642 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40582 2024-02-06T21:42:35.642 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.642 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.642 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.642 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.642 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-02-06T21:42:35.642 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.642 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.642 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.642 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.642 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.642 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_snapshots 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_snapshots' 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.643 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_snapshots 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_lock: /' 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.644 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.645 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.646 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.647 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.648 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.648 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.648 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40502/exe 2024-02-06T21:42:35.648 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.648 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.648 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.648 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.648 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.648 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.648 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.648 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.648 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.648 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.648 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.648 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.649 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.649 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.649 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.649 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.649 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.649 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.649 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.649 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.649 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.649 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.649 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.649 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.649 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.649 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.649 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-02-06T21:42:35.650 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2024-02-06T21:42:35.650 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_lock_pp: /' 2024-02-06T21:42:35.650 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.650 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.650 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.650 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.650 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.650 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.650 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.650 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.650 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.650 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.650 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.650 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.650 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.651 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.651 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.651 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.651 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.651 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.651 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.651 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.651 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.651 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.651 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.651 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.651 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.651 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_snapshots 2024-02-06T21:42:35.651 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40617 2024-02-06T21:42:35.651 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_snapshots on pid 40617' 2024-02-06T21:42:35.652 INFO:tasks.workunit.client.0.smithi120.stdout:test api_snapshots on pid 40617 2024-02-06T21:42:35.659 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40617 2024-02-06T21:42:35.659 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.659 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.659 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.660 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.660 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.660 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.660 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.660 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.660 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.660 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.660 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_snapshots_pp 2024-02-06T21:42:35.660 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.660 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.660 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.660 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.660 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_snapshots_pp' 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_snapshots_pp 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40526/exe 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.661 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.662 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.662 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.662 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.662 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.662 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.662 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.662 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.662 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.662 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.662 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.662 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.662 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.662 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.662 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.662 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.663 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.663 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.663 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.663 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.663 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.663 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.663 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.663 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.663 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.663 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.663 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.663 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.663 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.663 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.663 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.664 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.664 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.664 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.664 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.664 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_misc: /' 2024-02-06T21:42:35.664 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.664 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.664 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.664 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.664 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.664 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2024-02-06T21:42:35.664 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_misc.log 2024-02-06T21:42:35.664 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.666 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.666 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.666 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.666 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_snapshots_pp 2024-02-06T21:42:35.666 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40652 2024-02-06T21:42:35.666 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_snapshots_pp on pid 40652' 2024-02-06T21:42:35.666 INFO:tasks.workunit.client.0.smithi120.stdout:test api_snapshots_pp on pid 40652 2024-02-06T21:42:35.687 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40652 2024-02-06T21:42:35.687 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.687 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.687 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-02-06T21:42:35.687 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.687 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.687 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.687 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.687 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.687 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.687 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.687 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.687 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.688 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.688 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.688 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.688 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.688 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.688 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.688 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.688 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.688 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.688 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.688 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.688 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.688 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.688 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.689 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.690 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.691 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.692 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.692 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.692 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.692 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.692 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.692 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.692 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.692 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.692 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.692 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.692 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.692 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.692 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.692 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.692 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.693 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_stat 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.694 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.695 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.695 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.695 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.695 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.695 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.695 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.695 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.695 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.695 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.695 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.695 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.695 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.695 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.695 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.695 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.696 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.696 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.696 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.696 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.696 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.696 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.696 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.696 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.696 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.696 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.696 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.696 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.696 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.696 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.697 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.697 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.697 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.697 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.697 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.697 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.697 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.697 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_stat' 2024-02-06T21:42:35.697 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.697 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.697 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.697 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.697 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.697 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.697 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.698 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.699 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.699 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.699 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.699 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.699 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.699 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.699 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.699 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.699 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.699 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.699 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.699 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.699 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.699 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.699 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.700 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.700 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.700 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.700 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.700 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40617/exe 2024-02-06T21:42:35.700 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.700 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.700 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.700 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.700 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.700 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.700 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.700 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.700 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.701 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.701 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.701 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.701 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.701 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.701 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.701 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.701 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.701 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.701 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.701 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.701 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.701 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40558/exe 2024-02-06T21:42:35.701 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.701 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.702 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_snapshots: /' 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_stat 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-02-06T21:42:35.703 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2024-02-06T21:42:35.704 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.704 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_stat 2024-02-06T21:42:35.704 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40714 2024-02-06T21:42:35.704 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_stat on pid 40714' 2024-02-06T21:42:35.704 INFO:tasks.workunit.client.0.smithi120.stdout:test api_stat on pid 40714 2024-02-06T21:42:35.705 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40714 2024-02-06T21:42:35.705 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.705 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.705 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2024-02-06T21:42:35.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-02-06T21:42:35.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [----------] Global test environment set-up. 2024-02-06T21:42:35.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [----------] 12 tests from AsioRados 2024-02-06T21:42:35.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.585+0000 7fc1af40c780 10 monclient: get_monmap_and_config 2024-02-06T21:42:35.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.585+0000 7fc1af40c780 10 monclient: build_initial_monmap 2024-02-06T21:42:35.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.585+0000 7fc1af40c780 10 monclient: monmap: 2024-02-06T21:42:35.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: epoch 0 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: last_changed 0.000000 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: created 0.000000 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: min_mon_release 0 (unknown) 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: election_strategy: 1 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.585+0000 7fc1af40c780 1 Processor -- start 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.586+0000 7fc1af40c780 1 -- start start 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.586+0000 7fc1af40c780 10 monclient: init 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.586+0000 7fc1af40c780 10 monclient: _reopen_session rank -1 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.586+0000 7fc1af40c780 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.586+0000 7fc1af40c780 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ef778dfe60 0x55ef778e7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.587+0000 7fc1af40c780 10 monclient(hunting): picked mon.noname-b con 0x55ef778dfe60 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:42:35.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.587+0000 7fc1af40c780 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef778e6b30 0x55ef7797db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:42:35.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.587+0000 7fc1af40c780 10 monclient(hunting): picked mon.noname-a con 0x55ef778e6b30 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:42:35.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.587+0000 7fc1af40c780 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ef7797bda0 0x55ef778e96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:42:35.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.587+0000 7fc1af40c780 10 monclient(hunting): picked mon.noname-c con 0x55ef7797bda0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:42:35.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.587+0000 7fc1af40c780 10 monclient(hunting): start opening mon connection 2024-02-06T21:42:35.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.587+0000 7fc1af40c780 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef778601c0 con 0x55ef778e6b30 2024-02-06T21:42:35.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.587+0000 7fc1af40c780 10 monclient(hunting): start opening mon connection 2024-02-06T21:42:35.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.587+0000 7fc1af40c780 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef777f97c0 con 0x55ef778dfe60 2024-02-06T21:42:35.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.587+0000 7fc1af40c780 10 monclient(hunting): start opening mon connection 2024-02-06T21:42:35.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.587+0000 7fc1af40c780 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ef777f9460 con 0x55ef7797bda0 2024-02-06T21:42:35.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.587+0000 7fc1af40c780 10 monclient(hunting): _renew_subs 2024-02-06T21:42:35.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.587+0000 7fc1af40c780 10 monclient(hunting): authenticate will time out at 1385.924586s 2024-02-06T21:42:35.712 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_stat_pp 2024-02-06T21:42:35.712 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_stat_pp' 2024-02-06T21:42:35.712 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.712 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_tier_pp: /' 2024-02-06T21:42:35.713 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-02-06T21:42:35.713 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.713 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.713 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.713 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.713 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.713 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.713 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-02-06T21:42:35.713 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40582/exe 2024-02-06T21:42:35.713 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.713 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.713 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.713 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.713 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_stat_pp 2024-02-06T21:42:35.714 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_stat_pp 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40757 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.715 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.716 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.716 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.716 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.716 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.716 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.716 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.716 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.716 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_stat_pp on pid 40757' 2024-02-06T21:42:35.716 INFO:tasks.workunit.client.0.smithi120.stdout: test api_stat_pp on pid 40757 2024-02-06T21:42:35.727 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40757 2024-02-06T21:42:35.727 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.727 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.727 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.727 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.727 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-02-06T21:42:35.727 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_watch_notify 2024-02-06T21:42:35.728 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_watch_notify' 2024-02-06T21:42:35.728 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2024-02-06T21:42:35.728 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.728 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.728 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.728 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.728 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.728 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.728 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.728 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.728 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.728 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.728 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.728 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.729 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40545/exe 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_pool.log 2024-02-06T21:42:35.730 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.731 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.731 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.731 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.731 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.731 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.731 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.731 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.731 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.731 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.731 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.731 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.731 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.731 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.731 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.731 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.732 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.732 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.732 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.732 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.732 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.732 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.732 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.732 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.732 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.732 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.732 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.732 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_pool: /' 2024-02-06T21:42:35.732 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.732 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.732 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_misc_pp: /' 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40652/exe 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.733 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.734 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.735 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.736 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.736 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.736 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.736 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.736 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.736 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.736 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2024-02-06T21:42:35.736 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.736 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.736 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.736 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.736 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.736 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.736 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.736 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-02-06T21:42:35.737 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-02-06T21:42:35.737 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_watch_notify 2024-02-06T21:42:35.737 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_watch_notify 2024-02-06T21:42:35.737 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40791 2024-02-06T21:42:35.737 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_watch_notify on pid 40791' 2024-02-06T21:42:35.737 INFO:tasks.workunit.client.0.smithi120.stdout:test api_watch_notify on pid 40791 2024-02-06T21:42:35.740 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40791 2024-02-06T21:42:35.740 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.740 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.740 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_watch_notify_pp 2024-02-06T21:42:35.740 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_watch_notify_pp' 2024-02-06T21:42:35.740 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.740 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.740 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.740 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.740 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.740 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.740 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.740 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.741 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.741 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.741 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.741 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.741 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.741 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.741 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.741 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.741 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.741 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.741 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.741 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.741 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.741 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.741 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.589+0000 7fc1ac43a640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ef778dfe60 0x55ef778e7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:42:35.742 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.590+0000 7fc1ac43a640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ef778dfe60 0x55ef778e7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:51734/0 (socket says 172.21.15.120:51734) 2024-02-06T21:42:35.742 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.590+0000 7fc1ac43a640 1 -- 172.21.15.120:0/3774865752 learned_addr learned my addr 172.21.15.120:0/3774865752 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:42:35.742 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.590+0000 7fc1ac43a640 10 monclient(hunting): get_auth_request con 0x55ef778dfe60 auth_method 0 2024-02-06T21:42:35.742 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.590+0000 7fc1ac43a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:42:35.742 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.590+0000 7fc1ac43a640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:42:35.742 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.590+0000 7fc1ac43a640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:42:35.742 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.591+0000 7fc1ac43a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:42:35.742 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.591+0000 7fc1ac43a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:42:35.742 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.591+0000 7fc1ac43a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:42:35.742 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.599+0000 7fc1ac43a640 10 monclient(hunting): handle_auth_done global_id 4411 payload 274 2024-02-06T21:42:35.742 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.599+0000 7fc1ac43a640 1 -- 172.21.15.120:0/3774865752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ef7797bda0 msgr2=0x55ef778e96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:42:35.742 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.599+0000 7fc1ac43a640 1 --2- 172.21.15.120:0/3774865752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ef7797bda0 0x55ef778e96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:42:35.742 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.599+0000 7fc1ac43a640 1 -- 172.21.15.120:0/3774865752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef778e6b30 msgr2=0x55ef7797db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T21:42:35.742 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.599+0000 7fc1ac43a640 1 --2- 172.21.15.120:0/3774865752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef778e6b30 0x55ef7797db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:42:35.742 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.599+0000 7fc1ac43a640 10 monclient: _finish_hunting 0 2024-02-06T21:42:35.743 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.599+0000 7fc1ac43a640 1 monclient: found mon.noname-b 2024-02-06T21:42:35.743 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.599+0000 7fc1ac43a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:42:35.743 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.599+0000 7fc1ac43a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:42:35.743 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.599+0000 7fc1ac43a640 1 -- 172.21.15.120:0/3774865752 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a400a1c0 con 0x55ef778dfe60 2024-02-06T21:42:35.743 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.614+0000 7fc1ac43a640 1 --2- 172.21.15.120:0/3774865752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ef778dfe60 0x55ef778e7f10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00a5e0 tx=0x7fc19c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d73f068e78963df8 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:42:35.744 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.744 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.744 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.744 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.744 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.744 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-02-06T21:42:35.744 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.744 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.744 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.744 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.745 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.630+0000 7fc1aac37640 1 -- 172.21.15.120:0/3774865752 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c008a40 con 0x55ef778dfe60 2024-02-06T21:42:35.745 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.630+0000 7fc1aac37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:42:35.745 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.630+0000 7fc1aac37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:42:35.745 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.630+0000 7fc1aac37640 10 monclient: dump: 2024-02-06T21:42:35.745 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: epoch 1 2024-02-06T21:42:35.745 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:42:35.745 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:42:35.745 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:42:35.745 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: min_mon_release 19 (squid) 2024-02-06T21:42:35.745 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: election_strategy: 1 2024-02-06T21:42:35.745 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:42:35.745 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:42:35.745 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:42:35.746 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.746 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.630+0000 7fc1aac37640 10 monclient: _finish_auth 0 2024-02-06T21:42:35.746 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.630+0000 7fc1aac37640 10 monclient: _check_auth_tickets 2024-02-06T21:42:35.746 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.630+0000 7fc1aac37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:35.746 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.630+0000 7fc1af40c780 5 monclient: authenticate success, global_id 4411 2024-02-06T21:42:35.746 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.630+0000 7fc1af40c780 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T21:42:35.746 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.630+0000 7fc1aac37640 1 -- 172.21.15.120:0/3774865752 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19c008bc0 con 0x55ef778dfe60 2024-02-06T21:42:35.746 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.630+0000 7fc1aac37640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:42:35.746 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.630+0000 7fc1aac37640 1 -- 172.21.15.120:0/3774865752 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c013440 con 0x55ef778dfe60 2024-02-06T21:42:35.746 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.630+0000 7fc1af40c780 10 monclient: get_monmap_and_config success 2024-02-06T21:42:35.746 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.630+0000 7fc1af40c780 4 set_mon_vals no callback set 2024-02-06T21:42:35.746 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.631+0000 7fc1af40c780 10 monclient: shutdown 2024-02-06T21:42:35.746 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.631+0000 7fc1af40c780 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:42:35.746 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.631+0000 7fc1af40c780 1 -- 172.21.15.120:0/3774865752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ef778dfe60 msgr2=0x55ef778e7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:42:35.746 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.631+0000 7fc1af40c780 1 --2- 172.21.15.120:0/3774865752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ef778dfe60 0x55ef778e7f10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00a5e0 tx=0x7fc19c005f90 comp rx=0 tx=0).stop 2024-02-06T21:42:35.747 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.631+0000 7fc1aac37640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-02-06T21:42:35.747 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.631+0000 7fc1af40c780 1 -- 172.21.15.120:0/3774865752 shutdown_connections 2024-02-06T21:42:35.747 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.631+0000 7fc1af40c780 1 --2- 172.21.15.120:0/3774865752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ef7797bda0 0x55ef778e96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:42:35.749 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.749 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.749 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.749 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.749 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.749 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.749 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.750 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.750 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.750 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.750 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.750 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.750 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.750 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.750 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.750 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.750 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.750 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.750 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.750 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.750 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_watch_notify_pp 2024-02-06T21:42:35.751 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.751 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.751 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.751 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.751 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.751 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.751 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.751 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.631+0000 7fc1af40c780 1 --2- 172.21.15.120:0/3774865752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef778e6b30 0x55ef7797db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:42:35.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.631+0000 7fc1af40c780 1 --2- 172.21.15.120:0/3774865752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ef778dfe60 0x55ef778e7f10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:42:35.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.631+0000 7fc1af40c780 1 -- 172.21.15.120:0/3774865752 >> 172.21.15.120:0/3774865752 conn(0x55ef778a3c10 msgr2=0x55ef778e7590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:42:35.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.631+0000 7fc1af40c780 1 -- 172.21.15.120:0/3774865752 shutdown_connections 2024-02-06T21:42:35.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.631+0000 7fc1af40c780 1 -- 172.21.15.120:0/3774865752 wait complete. 2024-02-06T21:42:35.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.631+0000 7fc1af40c780 10 monclient: build_initial_monmap 2024-02-06T21:42:35.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.631+0000 7fc1af40c780 10 monclient: monmap: 2024-02-06T21:42:35.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: epoch 0 2024-02-06T21:42:35.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-02-06T21:42:35.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: last_changed 0.000000 2024-02-06T21:42:35.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: created 0.000000 2024-02-06T21:42:35.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: min_mon_release 0 (unknown) 2024-02-06T21:42:35.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: election_strategy: 1 2024-02-06T21:42:35.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T21:42:35.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T21:42:35.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T21:42:35.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.632+0000 7fc1af40c780 1 librados: starting msgr at 2024-02-06T21:42:35.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.632+0000 7fc1af40c780 1 librados: starting objecter 2024-02-06T21:42:35.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.632+0000 7fc1af40c780 1 Processor -- start 2024-02-06T21:42:35.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.632+0000 7fc1af40c780 1 -- start start 2024-02-06T21:42:35.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.632+0000 7fc1af40c780 1 librados: setting wanted keys 2024-02-06T21:42:35.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.632+0000 7fc1af40c780 1 librados: calling monclient init 2024-02-06T21:42:35.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.632+0000 7fc1af40c780 10 monclient: init 2024-02-06T21:42:35.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 7fc1af40c780 10 monclient: _reopen_session rank -1 2024-02-06T21:42:35.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 7fc1af40c780 10 monclient: _add_conns ranks=[0,1,2] 2024-02-06T21:42:35.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 7fc1af40c780 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef778dfe60 0x55ef778aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:42:35.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 7fc1af40c780 10 monclient(hunting): picked mon.noname-a con 0x55ef778dfe60 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T21:42:35.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 7fc1af40c780 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ef778e6b30 0x55ef77972e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:42:35.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 7fc1af40c780 10 monclient(hunting): picked mon.noname-b con 0x55ef778e6b30 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T21:42:35.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 7fc1af40c780 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ef7797bda0 0x55ef778a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:42:35.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 7fc1af40c780 10 monclient(hunting): picked mon.noname-c con 0x55ef7797bda0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T21:42:35.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 7fc1af40c780 10 monclient(hunting): start opening mon connection 2024-02-06T21:42:35.754 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 7fc1af40c780 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef777d8550 con 0x55ef778dfe60 2024-02-06T21:42:35.754 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.754 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_watch_notify_pp 2024-02-06T21:42:35.754 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40825 2024-02-06T21:42:35.754 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_watch_notify_pp on pid 40825' 2024-02-06T21:42:35.755 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 test api_watch_notify_pp on pid 40825 2024-02-06T21:42:35.764 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40825 2024-02-06T21:42:35.764 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.764 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.764 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_cmd 2024-02-06T21:42:35.764 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_cmd' 2024-02-06T21:42:35.764 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.764 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.764 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.764 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_cmd 2024-02-06T21:42:35.764 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.764 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.764 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.764 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.764 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40714/exe 2024-02-06T21:42:35.765 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.766 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.767 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.767 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.767 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.767 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.767 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.767 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.767 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.767 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.767 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.767 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.767 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.767 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.767 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.767 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.767 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.768 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.768 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.768 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.768 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.768 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.768 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_stat.log 2024-02-06T21:42:35.768 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2024-02-06T21:42:35.768 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_stat: /' 2024-02-06T21:42:35.768 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.768 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.768 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.768 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.768 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.768 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.768 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.769 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.769 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.769 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.769 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.769 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.769 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.769 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.769 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.769 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.769 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.769 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.769 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-02-06T21:42:35.769 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.769 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.769 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.770 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.770 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.770 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.770 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.770 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.770 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.770 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.770 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.770 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.770 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.770 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.770 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.770 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.770 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.770 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.771 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.771 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.771 INFO:tasks.workunit.client.0.smithi120.stdout:7fc1af40c780 10 monclient(hunting): start opening mon connection 2024-02-06T21:42:35.771 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 7fc1af40c780 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef778aaa10 con 0x55ef778e6b30 2024-02-06T21:42:35.771 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 7fc1af40c780 10 monclient(hunting): start opening mon connection 2024-02-06T21:42:35.771 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 7fc1af40c780 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ef778a9dd0 con 0x55ef7797bda0 2024-02-06T21:42:35.771 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 7fc1af40c780 10 monclient(hunting): _renew_subs 2024-02-06T21:42:35.771 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.633+0000 7fc1af40c780 10 monclient(hunting): authenticate will time out at 1385.971159s 2024-02-06T21:42:35.771 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.642+0000 7fc1ac43a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef778dfe60 0x55ef778aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:42:35.771 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.648+0000 7fc1abc39640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ef778e6b30 0x55ef77972e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:42:35.771 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.648+0000 7fc1acc3b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ef7797bda0 0x55ef778a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:42:35.771 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.649+0000 7fc1abc39640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ef778e6b30 0x55ef77972e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:51786/0 (socket says 172.21.15.120:51786) 2024-02-06T21:42:35.772 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.649+0000 7fc1abc39640 1 -- 172.21.15.120:0/2499202281 learned_addr learned my addr 172.21.15.120:0/2499202281 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T21:42:35.772 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.649+0000 7fc1acc3b640 10 monclient(hunting): get_auth_request con 0x55ef7797bda0 auth_method 0 2024-02-06T21:42:35.772 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.649+0000 7fc1acc3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:42:35.772 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.649+0000 7fc1acc3b640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:42:35.772 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.649+0000 7fc1acc3b640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:42:35.772 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.649+0000 7fc1abc39640 10 monclient(hunting): get_auth_request con 0x55ef778e6b30 auth_method 0 2024-02-06T21:42:35.772 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.649+0000 7fc1abc39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:42:35.772 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.649+0000 7fc1abc39640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:42:35.772 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.649+0000 7fc1abc39640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:42:35.772 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.652+0000 7fc1ac43a640 10 monclient(hunting): get_auth_request con 0x55ef778dfe60 auth_method 0 2024-02-06T21:42:35.772 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.652+0000 7fc1ac43a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T21:42:35.772 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.652+0000 7fc1ac43a640 10 monclient(hunting): _init_auth method 2 2024-02-06T21:42:35.772 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.652+0000 7fc1ac43a640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T21:42:35.772 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.653+0000 7fc1abc39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:42:35.772 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.653+0000 7fc1abc39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:42:35.777 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.777 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.777 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.777 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.777 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.777 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.777 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.777 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.777 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.777 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40757/exe 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.780 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.780 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.780 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2024-02-06T21:42:35.780 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_stat_pp: /' 2024-02-06T21:42:35.780 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-02-06T21:42:35.780 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_cmd 2024-02-06T21:42:35.780 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40875 2024-02-06T21:42:35.780 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_cmd on pid 40875' 2024-02-06T21:42:35.780 INFO:tasks.workunit.client.0.smithi120.stdout: test api_cmd on pid 40875 2024-02-06T21:42:35.783 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40875 2024-02-06T21:42:35.783 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.783 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.783 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_cmd_pp 2024-02-06T21:42:35.783 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_cmd_pp' 2024-02-06T21:42:35.783 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_cmd_pp 2024-02-06T21:42:35.783 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.783 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.783 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.783 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.783 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.783 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.783 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.783 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_cmd_pp 2024-02-06T21:42:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40901 2024-02-06T21:42:35.785 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_cmd_pp on pid 40901' 2024-02-06T21:42:35.785 INFO:tasks.workunit.client.0.smithi120.stdout:test api_cmd_pp on pid 40901 2024-02-06T21:42:35.789 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40901 2024-02-06T21:42:35.790 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.790 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.790 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-02-06T21:42:35.790 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.790 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.790 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.790 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.790 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.790 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.790 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_service 2024-02-06T21:42:35.790 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_service' 2024-02-06T21:42:35.790 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.790 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.790 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.790 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.793 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.794 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.794 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.794 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.794 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.794 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.794 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.794 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.794 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.794 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.794 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.794 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.794 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.794 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.653+0000 7fc1abc39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:42:35.795 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.655+0000 7fc1acc3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:42:35.795 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.655+0000 7fc1acc3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:42:35.795 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.655+0000 7fc1acc3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:42:35.795 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.655+0000 7fc1ac43a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T21:42:35.795 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.655+0000 7fc1ac43a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T21:42:35.795 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.655+0000 7fc1ac43a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T21:42:35.795 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.660+0000 7fc1abc39640 10 monclient(hunting): handle_auth_done global_id 4426 payload 824 2024-02-06T21:42:35.795 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.660+0000 7fc1abc39640 1 -- 172.21.15.120:0/2499202281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ef7797bda0 msgr2=0x55ef778a6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:42:35.795 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.660+0000 7fc1abc39640 1 --2- 172.21.15.120:0/2499202281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ef7797bda0 0x55ef778a6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:42:35.795 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.660+0000 7fc1abc39640 1 -- 172.21.15.120:0/2499202281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef778dfe60 msgr2=0x55ef778aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:42:35.795 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.660+0000 7fc1abc39640 1 --2- 172.21.15.120:0/2499202281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef778dfe60 0x55ef778aa4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:42:35.795 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.660+0000 7fc1abc39640 10 monclient: _finish_hunting 0 2024-02-06T21:42:35.795 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.660+0000 7fc1abc39640 1 monclient: found mon.noname-b 2024-02-06T21:42:35.796 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.660+0000 7fc1abc39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:42:35.796 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.660+0000 7fc1abc39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T21:42:35.796 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.660+0000 7fc1abc39640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c005d70 con 0x55ef778e6b30 2024-02-06T21:42:35.796 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.663+0000 7fc1abc39640 1 --2- 172.21.15.120:0/2499202281 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ef778e6b30 0x55ef77972e30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000af50 tx=0x7fc1a00087f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:42:35.796 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.663+0000 7fc1a9c35640 10 client.?.objecter ms_handle_connect 0x55ef778e6b30 2024-02-06T21:42:35.796 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.663+0000 7fc1a9c35640 10 client.?.objecter resend_mon_ops 2024-02-06T21:42:35.796 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.680+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a000b860 con 0x55ef778e6b30 2024-02-06T21:42:35.796 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.680+0000 7fc1a9c35640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:42:35.796 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.680+0000 7fc1a9c35640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T21:42:35.796 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.680+0000 7fc1a9c35640 10 monclient: dump: 2024-02-06T21:42:35.796 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: epoch 1 2024-02-06T21:42:35.799 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.799 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_service 2024-02-06T21:42:35.799 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.799 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.800 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.800 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.800 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.800 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.800 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.800 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.800 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.800 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.800 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.800 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.800 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.800 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.800 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.800 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_service 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40921 2024-02-06T21:42:35.801 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_service on pid 40921' 2024-02-06T21:42:35.802 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7ctest api_service on pid 40921 2024-02-06T21:42:35.812 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40921 2024-02-06T21:42:35.812 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_service_pp 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_service_pp' 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.815 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_service_pp 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.816 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.817 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.818 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.819 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_service_pp 2024-02-06T21:42:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.821 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.821 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.821 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.821 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40953 2024-02-06T21:42:35.821 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.821 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.821 INFO:tasks.workunit.client.0.smithi120.stdout:test api_service_pp on pid 40953 2024-02-06T21:42:35.826 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_service_pp on pid 40953' 2024-02-06T21:42:35.826 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40953 2024-02-06T21:42:35.826 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.826 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.826 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.826 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_c_write_operations 2024-02-06T21:42:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_c_write_operations' 2024-02-06T21:42:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-02-06T21:42:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_c_write_operations 2024-02-06T21:42:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.830 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.830 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.830 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.830 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.830 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.830 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.830 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.830 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.830 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.830 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.830 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.830 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.830 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.830 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.830 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.831 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.831 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_c_write_operations 2024-02-06T21:42:35.831 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40968 2024-02-06T21:42:35.831 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_c_write_operations on pid 40968' 2024-02-06T21:42:35.831 INFO:tasks.workunit.client.0.smithi120.stdout:test api_c_write_operations on pid 40968 2024-02-06T21:42:35.847 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-02-06T21:42:35.847 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.847 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=40968 2024-02-06T21:42:35.847 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.847 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.848 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.848 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.848 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.848 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.848 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.848 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.848 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.848 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.848 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_c_read_operations 2024-02-06T21:42:35.848 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.848 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.848 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.848 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.848 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40791/exe 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_c_read_operations' 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.849 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.850 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.851 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.852 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.852 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.852 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.852 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.852 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.852 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.852 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.852 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.852 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.852 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.852 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.852 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.852 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.853 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.854 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.854 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.854 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.854 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.854 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.854 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.854 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.854 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.854 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.854 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.854 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.854 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.854 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.854 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.854 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.855 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.855 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.855 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.855 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.855 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.855 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40901/exe 2024-02-06T21:42:35.855 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.855 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.855 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.855 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.855 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.855 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.855 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.855 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.855 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.856 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.856 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.856 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.856 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.856 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.856 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_c_read_operations 2024-02-06T21:42:35.856 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.856 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.856 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.856 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.856 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.856 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2024-02-06T21:42:35.856 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.856 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.857 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_watch_notify: /' 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.858 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.859 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.860 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.860 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.860 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.860 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.860 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.860 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.860 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.860 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.860 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.860 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.860 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.860 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.860 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.860 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.860 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.861 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_c_read_operations 2024-02-06T21:42:35.861 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.861 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.861 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.861 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41005 2024-02-06T21:42:35.861 INFO:tasks.workunit.client.0.smithi120.stdout:test api_c_read_operations on pid 41005 2024-02-06T21:42:35.869 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_c_read_operations on pid 41005' 2024-02-06T21:42:35.869 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41005 2024-02-06T21:42:35.869 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-02-06T21:42:35.869 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.869 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.869 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.869 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.869 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.869 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.869 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.869 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40825/exe 2024-02-06T21:42:35.869 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.869 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.869 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.870 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.871 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.872 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.872 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.872 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.872 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.872 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.872 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.872 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.872 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.872 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.872 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.872 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.872 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.872 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.872 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.872 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s api_cls_remote_reads 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' api_cls_remote_reads' 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2024-02-06T21:42:35.873 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo api_cls_remote_reads 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40875/exe 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.874 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.875 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.875 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.875 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.875 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.875 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.875 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.875 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.875 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.875 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.875 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40921/exe 2024-02-06T21:42:35.875 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.875 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=api_cls_remote_reads 2024-02-06T21:42:35.875 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41029 2024-02-06T21:42:35.875 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test api_cls_remote_reads on pid 41029' 2024-02-06T21:42:35.876 INFO:tasks.workunit.client.0.smithi120.stdout:test api_cls_remote_reads on pid 41029 2024-02-06T21:42:35.876 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41029 2024-02-06T21:42:35.876 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.876 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.876 INFO:tasks.workunit.client.0.smithi120.stdout: 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: min_mon_release 19 (squid) 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: election_strategy: 1 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.680+0000 7fc1a9c35640 10 monclient: _finish_auth 0 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.680+0000 7fc1a9c35640 10 monclient: _check_auth_tickets 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.680+0000 7fc1a9c35640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.680+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a000b9e0 con 0x55ef778e6b30 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.680+0000 7fc1a9c35640 10 monclient: handle_config config(0 keys) v1 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.680+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a000bd90 con 0x55ef778e6b30 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.680+0000 7fc1a9c35640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.680+0000 7fc1a9c35640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T21:42:35.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.680+0000 7fc1a9c35640 10 monclient: dump: 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: epoch 1 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: created 2024-02-06T21:39:47.348445+0000 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: min_mon_release 19 (squid) 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: election_strategy: 1 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.680+0000 7fc1aa436640 4 set_mon_vals no callback set 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.683+0000 7fc1af40c780 5 monclient: authenticate success, global_id 4426 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.683+0000 7fc1af40c780 10 monclient: _renew_subs 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.683+0000 7fc1af40c780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.683+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ef778051f0 con 0x55ef778e6b30 2024-02-06T21:42:35.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.683+0000 7fc1af40c780 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:42:35.879 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.683+0000 7fc1af40c780 10 monclient: _renew_subs 2024-02-06T21:42:35.879 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.683+0000 7fc1af40c780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:42:35.879 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.683+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef777f83a0 con 0x55ef778e6b30 2024-02-06T21:42:35.879 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.683+0000 7fc1af40c780 1 librados: init done 2024-02-06T21:42:35.879 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.683+0000 7fc1af40c780 10 librados: wait_for_osdmap waiting 2024-02-06T21:42:35.879 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.716+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc1a0011070 con 0x55ef778e6b30 2024-02-06T21:42:35.891 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.891 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.891 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-02-06T21:42:35.891 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.891 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.891 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.891 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.891 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.891 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s list_parallel 2024-02-06T21:42:35.891 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' list_parallel' 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.892 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo list_parallel 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.893 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_service.log 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2024-02-06T21:42:35.894 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_service: /' 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.895 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.896 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.897 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.897 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.897 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.897 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.897 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.897 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.897 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_cmd.log 2024-02-06T21:42:35.897 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_cmd: /' 2024-02-06T21:42:35.897 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.897 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.897 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.897 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.897 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.897 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.897 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.898 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.899 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.899 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.899 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.899 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.899 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.899 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.899 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.899 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.899 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.899 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.899 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.899 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.899 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.899 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.899 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.900 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.900 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.900 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.900 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.900 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.900 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.900 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.900 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.900 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.900 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=list_parallel 2024-02-06T21:42:35.900 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.900 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.900 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.900 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.900 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.901 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.901 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41077 2024-02-06T21:42:35.901 INFO:tasks.workunit.client.0.smithi120.stdout: test list_parallel on pid 41077 2024-02-06T21:42:35.904 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test list_parallel on pid 41077' 2024-02-06T21:42:35.904 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41077 2024-02-06T21:42:35.904 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.904 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.904 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s open_pools_parallel 2024-02-06T21:42:35.904 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' open_pools_parallel' 2024-02-06T21:42:35.904 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo open_pools_parallel 2024-02-06T21:42:35.904 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.905 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.905 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.905 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.905 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.905 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.905 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.905 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-02-06T21:42:35.905 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.905 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.905 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.905 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.905 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.905 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.905 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.905 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.906 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.906 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.906 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.906 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.906 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.906 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.906 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.906 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.906 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.906 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.906 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.906 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.906 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.906 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=open_pools_parallel 2024-02-06T21:42:35.906 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41086 2024-02-06T21:42:35.907 INFO:tasks.workunit.client.0.smithi120.stdout:test open_pools_parallel on pid 41086 2024-02-06T21:42:35.914 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test open_pools_parallel on pid 41086' 2024-02-06T21:42:35.914 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41086 2024-02-06T21:42:35.915 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-06T21:42:35.915 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.915 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.915 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.915 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.915 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.915 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.915 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.915 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.915 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.915 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.915 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.915 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.915 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.915 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40953/exe 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s delete_pools_parallel 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' delete_pools_parallel' 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.916 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.917 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo delete_pools_parallel 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.918 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.919 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.920 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.921 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.921 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=delete_pools_parallel 2024-02-06T21:42:35.921 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41105 2024-02-06T21:42:35.921 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test delete_pools_parallel on pid 41105' 2024-02-06T21:42:35.921 INFO:tasks.workunit.client.0.smithi120.stdout:test delete_pools_parallel on pid 41105 2024-02-06T21:42:35.932 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41105 2024-02-06T21:42:35.932 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s cls 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' cls' 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo cls 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.933 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.934 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.934 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.934 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.934 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.934 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.934 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.934 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.934 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.934 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.934 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.934 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.934 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.934 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.934 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.934 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_service_pp: /' 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.935 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.936 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.936 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.936 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.936 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.936 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.936 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.936 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.936 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.936 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.936 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.936 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.936 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.936 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.936 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.936 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.937 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.937 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.937 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.937 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.937 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.937 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.937 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.937 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.937 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.937 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.937 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.937 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.937 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.937 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.937 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.938 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.940 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.940 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.940 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.940 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.940 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.940 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.940 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.940 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.940 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.940 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.940 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.940 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.940 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=cls 2024-02-06T21:42:35.940 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41140 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.942 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.942 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.942 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.942 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.942 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.942 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.942 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.942 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.942 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.942 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.942 INFO:tasks.workunit.client.0.smithi120.stdout:test cls on pid 41140 2024-02-06T21:42:35.952 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test cls on pid 41140' 2024-02-06T21:42:35.952 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41140 2024-02-06T21:42:35.952 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-06T21:42:35.952 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.952 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.952 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.952 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.952 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s cmd 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' cmd' 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.953 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo cmd 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41029/exe 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.954 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.955 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.956 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.956 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.956 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.956 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.956 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.956 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.956 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.956 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.956 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.956 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41005/exe 2024-02-06T21:42:35.956 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.956 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.956 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.956 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.956 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.957 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.958 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.958 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.958 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.958 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.958 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.958 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.958 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.958 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.958 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.958 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.958 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.958 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.958 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.958 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.958 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.959 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/40968/exe 2024-02-06T21:42:35.960 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.960 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.960 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.960 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-02-06T21:42:35.960 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.960 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.960 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-02-06T21:42:35.960 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=cmd 2024-02-06T21:42:35.960 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41176 2024-02-06T21:42:35.960 INFO:tasks.workunit.client.0.smithi120.stdout:test cmd on pid 41176 2024-02-06T21:42:35.976 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test cmd on pid 41176' 2024-02-06T21:42:35.976 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41176 2024-02-06T21:42:35.977 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-06T21:42:35.977 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:35.977 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.977 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.977 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.977 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.977 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.977 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.977 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.977 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.977 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.977 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.977 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.977 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.978 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.978 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.978 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.978 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.978 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.978 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.978 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.978 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.978 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2024-02-06T21:42:35.978 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.978 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.978 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.978 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.978 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.978 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.979 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.980 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.980 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.980 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.980 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.980 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.980 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.980 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.980 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-02-06T21:42:35.980 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.980 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.980 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.980 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.980 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.980 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.980 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s handler_error 2024-02-06T21:42:35.981 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' handler_error' 2024-02-06T21:42:35.981 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:35.981 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:35.981 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:35.981 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:35.981 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.981 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:35.981 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:35.981 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:35.981 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:35.981 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.981 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:35.981 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.981 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.982 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.983 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.984 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.984 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.984 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.984 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.984 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.984 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.984 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.984 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.984 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.984 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.984 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.984 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.984 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.984 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:35.985 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:35.986 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:35.987 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:35.987 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:35.987 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:35.987 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:35.987 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.987 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:35.987 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.987 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:35.987 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:35.987 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:35.987 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:35.987 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.987 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:35.987 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.987 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:35.988 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:35.988 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:35.988 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:35.988 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:35.988 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.988 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:35.988 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.988 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:35.988 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.988 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.988 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:35.988 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.988 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:35.988 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41086/exe 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:35.989 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo handler_error 2024-02-06T21:42:35.990 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-02-06T21:42:35.991 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=handler_error 2024-02-06T21:42:35.991 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-02-06T21:42:35.991 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41225 2024-02-06T21:42:35.991 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test handler_error on pid 41225' 2024-02-06T21:42:35.991 INFO:tasks.workunit.client.0.smithi120.stdout:test handler_error on pid 41225 2024-02-06T21:42:36.001 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41225 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s io 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:36.002 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:36.003 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:36.004 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' io' 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:36.005 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:36.006 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.007 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:36.008 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.008 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:36.008 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:36.008 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:36.008 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:36.008 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:36.008 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.008 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:36.008 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:36.008 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.008 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:36.008 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:36.008 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.008 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.009 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:36.009 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.009 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:36.009 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.009 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:36.009 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:36.009 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.009 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:36.009 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.009 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:36.009 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:36.009 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41105/exe 2024-02-06T21:42:36.009 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo io 2024-02-06T21:42:36.009 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:36.009 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41077/exe 2024-02-06T21:42:36.010 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=io 2024-02-06T21:42:36.010 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:36.010 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:36.010 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41260 2024-02-06T21:42:36.010 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:36.010 INFO:tasks.workunit.client.0.smithi120.stdout:test io on pid 41260 2024-02-06T21:42:36.021 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test io on pid 41260' 2024-02-06T21:42:36.021 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41260 2024-02-06T21:42:36.021 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-06T21:42:36.021 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:36.021 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:36.021 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:36.021 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:36.021 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:36.021 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:36.021 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:36.021 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:36.021 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:36.021 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:36.021 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_list_parallel.log 2024-02-06T21:42:36.022 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ list_parallel: /' 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.023 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:36.024 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.024 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:36.024 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:36.024 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:36.024 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:36.024 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.024 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:36.024 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.024 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:36.024 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.024 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:36.024 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:36.024 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:36.024 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s ec_io 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' ec_io' 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo ec_io 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.029 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.029 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:36.029 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.029 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:36.029 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.029 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-02-06T21:42:36.029 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-02-06T21:42:36.029 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:36.029 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=ec_io 2024-02-06T21:42:36.029 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41333 2024-02-06T21:42:36.029 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test ec_io on pid 41333' 2024-02-06T21:42:36.029 INFO:tasks.workunit.client.0.smithi120.stdout:test ec_io on pid 41333 2024-02-06T21:42:36.041 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41333 2024-02-06T21:42:36.041 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-06T21:42:36.041 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:36.041 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41176/exe 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41140/exe 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:36.042 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:36.043 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:36.043 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:36.043 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:36.043 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:36.043 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:36.043 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_neorados_cls 2024-02-06T21:42:36.043 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:36.043 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:36.043 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:36.043 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:36.043 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:36.043 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:36.043 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:36.043 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:36.043 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:36.044 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:36.044 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:36.044 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_neorados_cls.log 2024-02-06T21:42:36.044 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_neorados_cmd.log 2024-02-06T21:42:36.044 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_neorados_cmd 2024-02-06T21:42:36.044 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ cmd: /' 2024-02-06T21:42:36.044 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ cls: /' 2024-02-06T21:42:36.044 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:36.044 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:36.044 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:36.044 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.044 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:36.044 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.044 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:36.044 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:36.045 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:36.046 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:36.046 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.046 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:36.046 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:36.046 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:36.046 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:36.046 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.046 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:36.046 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.046 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:36.046 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:36.046 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.046 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:36.046 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.046 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:36.047 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.047 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.047 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.047 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:36.047 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.047 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:36.047 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.047 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:36.047 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:36.047 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:36.047 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.047 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:36.047 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.047 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:36.047 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s list 2024-02-06T21:42:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' list' 2024-02-06T21:42:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo list 2024-02-06T21:42:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:36.050 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.050 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:36.050 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:36.050 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:36.050 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:36.050 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:36.050 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.050 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:36.050 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.050 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:36.050 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:36.050 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.050 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:36.050 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:36.050 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=list 2024-02-06T21:42:36.051 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41410 2024-02-06T21:42:36.051 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test list on pid 41410' 2024-02-06T21:42:36.051 INFO:tasks.workunit.client.0.smithi120.stdout:test list on pid 41410 2024-02-06T21:42:36.054 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41410 2024-02-06T21:42:36.054 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-06T21:42:36.054 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:36.054 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-02-06T21:42:36.054 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s ec_list 2024-02-06T21:42:36.054 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' ec_list' 2024-02-06T21:42:36.054 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:36.054 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:36.055 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:36.055 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.055 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:36.055 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.055 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:36.055 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:36.055 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.055 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:36.055 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.055 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:36.055 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo ec_list 2024-02-06T21:42:36.055 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.055 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:36.055 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.055 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:36.056 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:36.056 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:36.056 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.056 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:36.056 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.056 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:36.056 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:36.056 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:36.056 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=ec_list 2024-02-06T21:42:36.056 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41432 2024-02-06T21:42:36.056 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test ec_list on pid 41432' 2024-02-06T21:42:36.056 INFO:tasks.workunit.client.0.smithi120.stdout:test ec_list on pid 41432 2024-02-06T21:42:36.061 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41432 2024-02-06T21:42:36.061 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-06T21:42:36.061 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:36.061 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s misc 2024-02-06T21:42:36.061 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' misc' 2024-02-06T21:42:36.061 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:36.061 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-02-06T21:42:36.061 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:36.061 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:36.061 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo misc 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:36.064 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=misc 2024-02-06T21:42:36.064 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41458 2024-02-06T21:42:36.064 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test misc on pid 41458' 2024-02-06T21:42:36.064 INFO:tasks.workunit.client.0.smithi120.stdout:test misc on pid 41458 2024-02-06T21:42:36.068 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41458 2024-02-06T21:42:36.068 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-06T21:42:36.068 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:36.068 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-02-06T21:42:36.068 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:36.068 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:36.068 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:36.068 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:36.068 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.068 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:36.068 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s pool 2024-02-06T21:42:36.068 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' pool' 2024-02-06T21:42:36.068 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo pool 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.069 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=pool 2024-02-06T21:42:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41481 2024-02-06T21:42:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:36.071 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test pool on pid 41481' 2024-02-06T21:42:36.071 INFO:tasks.workunit.client.0.smithi120.stdout:test pool on pid 41481 2024-02-06T21:42:36.083 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.083 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41481 2024-02-06T21:42:36.083 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-06T21:42:36.084 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:36.084 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-02-06T21:42:36.084 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.084 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s read_operations 2024-02-06T21:42:36.084 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:36.084 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.084 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:36.084 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:36.084 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' read_operations' 2024-02-06T21:42:36.084 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:36.084 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.084 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:36.084 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:36.084 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41225/exe 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.085 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.086 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.087 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:36.088 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.088 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:36.088 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.088 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:36.088 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:36.088 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:36.088 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:36.088 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:36.088 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:36.088 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:36.088 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:36.088 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:36.088 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:36.088 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:36.088 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_neorados_handler_error 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.089 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:36.090 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.090 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:36.090 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:36.090 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:36.090 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.090 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:36.090 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.090 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:36.090 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.090 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.090 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:36.090 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.090 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:36.090 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:36.090 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_neorados_handler_error.log 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:36.091 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.092 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:36.092 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:36.092 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ handler_error: /' 2024-02-06T21:42:36.092 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:36.092 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:36.092 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:36.092 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:36.092 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.092 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:36.092 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.092 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:36.092 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:36.092 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.092 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.092 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo read_operations 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.093 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:36.094 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=read_operations 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:36.095 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.096 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:36.096 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41522 2024-02-06T21:42:36.096 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.096 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:36.096 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:36.096 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test read_operations on pid 41522' 2024-02-06T21:42:36.096 INFO:tasks.workunit.client.0.smithi120.stdout:test read_operations on pid 41522 2024-02-06T21:42:36.110 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41522 2024-02-06T21:42:36.110 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-06T21:42:36.110 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:36.110 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41260/exe 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.111 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:36.112 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41333/exe 2024-02-06T21:42:36.112 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:36.112 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:36.112 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.112 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:36.112 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.112 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:36.112 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:36.112 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:36.112 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.112 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:36.112 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.112 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:36.112 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s snapshots 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.113 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' snapshots' 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:36.114 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_neorados_io 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:36.115 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:36.116 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.116 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:36.116 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:36.116 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:36.116 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.116 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:36.116 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:36.116 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:36.116 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.116 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:36.116 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.116 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:36.116 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.116 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:36.116 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ io: /' 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_neorados_io.log 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_neorados_ec_io 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_neorados_ec_io.log 2024-02-06T21:42:36.117 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.118 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:36.118 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.118 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:36.118 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:36.118 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ ec_io: /' 2024-02-06T21:42:36.118 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo snapshots 2024-02-06T21:42:36.118 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:36.118 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:36.118 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:36.118 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:36.118 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.118 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:36.118 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.118 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:36.118 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.119 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:36.120 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:36.120 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:36.120 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:36.120 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.120 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:36.120 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.120 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:36.120 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:36.120 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:36.120 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.120 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.120 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:36.120 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.120 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.120 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.121 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:36.122 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.122 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:36.122 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:36.122 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:36.122 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:36.122 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.122 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:36.122 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:36.122 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.122 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.122 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.122 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:36.122 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.122 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:36.122 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:36.123 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:36.123 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.123 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:36.123 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.123 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:36.123 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:36.123 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.123 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:36.123 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:36.123 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.123 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:36.123 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:36.123 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=snapshots 2024-02-06T21:42:36.123 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41558 2024-02-06T21:42:36.123 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test snapshots on pid 41558' 2024-02-06T21:42:36.124 INFO:tasks.workunit.client.0.smithi120.stdout:test snapshots on pid 41558 2024-02-06T21:42:36.133 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41558 2024-02-06T21:42:36.133 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-06T21:42:36.133 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:36.133 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s watch_notify 2024-02-06T21:42:36.133 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:36.133 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:36.133 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:36.133 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.133 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:36.133 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.133 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:36.133 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:36.133 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41410/exe 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' watch_notify' 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:36.134 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.135 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:36.135 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:36.135 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-02-06T21:42:36.135 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.135 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.135 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:36.135 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.135 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:36.135 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.135 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:36.135 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:36.135 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.135 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:36.135 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.135 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:36.136 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:36.136 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:36.136 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:36.136 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:36.136 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:36.136 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:36.136 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:36.136 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:36.136 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:36.136 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:36.136 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:36.136 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:36.136 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:36.136 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:36.136 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.137 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41481/exe 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_neorados_list.log 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:36.138 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo watch_notify 2024-02-06T21:42:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ list: /' 2024-02-06T21:42:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_neorados_pool 2024-02-06T21:42:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=watch_notify 2024-02-06T21:42:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41581 2024-02-06T21:42:36.141 INFO:tasks.workunit.client.0.smithi120.stdout:test watch_notify on pid 41581 2024-02-06T21:42:36.154 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test watch_notify on pid 41581' 2024-02-06T21:42:36.154 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41581 2024-02-06T21:42:36.155 INFO:tasks.workunit.client.0.smithi120.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-06T21:42:36.155 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:36.155 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-02-06T21:42:36.155 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_neorados_list 2024-02-06T21:42:36.155 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:36.155 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:36.155 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:36.155 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.155 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:36.155 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.155 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:36.155 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:36.155 INFO:tasks.workunit.client.0.smithi120.stderr:++ printf %25s write_operations 2024-02-06T21:42:36.155 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:36.155 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.156 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:36.156 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.156 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:36.156 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:36.156 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:36.156 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.156 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:36.156 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.156 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:36.156 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.156 INFO:tasks.workunit.client.0.smithi120.stderr:+ r=' write_operations' 2024-02-06T21:42:36.156 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ pool: /' 2024-02-06T21:42:36.156 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.156 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:36.157 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.157 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:36.157 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:36.157 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_neorados_pool.log 2024-02-06T21:42:36.157 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:36.157 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:36.157 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:36.157 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:36.157 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.157 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:36.157 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:36.157 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:36.157 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.157 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:36.157 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:36.158 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:36.159 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.160 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:36.160 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:36.160 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:36.160 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:36.160 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:36.160 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:36.160 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.160 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:36.160 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.160 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:36.160 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.160 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:36.160 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.160 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.161 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41458/exe 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:36.162 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:36.163 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:++ echo write_operations 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_neorados_misc 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ misc: /' 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:++ awk '{print $1}' 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:36.164 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:36.165 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.165 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:36.165 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.165 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:36.165 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:36.165 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.165 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:36.165 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:36.165 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:36.165 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.165 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:36.165 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.165 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:36.165 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:36.165 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.166 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.166 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:36.166 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.166 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:36.166 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.166 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:36.166 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:36.166 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.166 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:36.166 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.166 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:36.166 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:36.166 INFO:tasks.workunit.client.0.smithi120.stderr:+ ff=write_operations 2024-02-06T21:42:36.166 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41625 2024-02-06T21:42:36.166 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:36.167 INFO:tasks.workunit.client.0.smithi120.stdout:test write_operations on pid 41625 2024-02-06T21:42:36.192 INFO:tasks.workunit.client.0.smithi120.stderr:+ echo 'test write_operations on pid 41625' 2024-02-06T21:42:36.192 INFO:tasks.workunit.client.0.smithi120.stderr:+ pids[$f]=41625 2024-02-06T21:42:36.192 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:36.192 INFO:tasks.workunit.client.0.smithi120.stderr:+ ret=0 2024-02-06T21:42:36.192 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:36.192 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' 1 -eq 1 ']' 2024-02-06T21:42:36.192 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.192 INFO:tasks.workunit.client.0.smithi120.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-02-06T21:42:36.192 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:36.192 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.192 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:36.193 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T21:42:36.193 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:36.193 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40558 2024-02-06T21:42:36.193 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40558 2024-02-06T21:42:36.193 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.193 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:36.193 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:36.193 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:36.195 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_neorados_misc.log 2024-02-06T21:42:36.195 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.195 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:36.195 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.195 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:36.195 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:36.195 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.195 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:36.195 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.195 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:36.195 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.195 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:36.195 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.195 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:36.196 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:36.196 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:36.196 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -f /etc/bashrc ']' 2024-02-06T21:42:36.196 INFO:tasks.workunit.client.0.smithi120.stderr:+ . /etc/bashrc 2024-02-06T21:42:36.196 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -z '' ']' 2024-02-06T21:42:36.196 INFO:tasks.workunit.client.0.smithi120.stderr:++ BASHRCSOURCED=Y 2024-02-06T21:42:36.196 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.196 INFO:tasks.workunit.client.0.smithi120.stderr:++ shopt -q login_shell 2024-02-06T21:42:36.196 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:36.196 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:36.196 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:36.196 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.196 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:36.196 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.196 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:36.197 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:36.197 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.197 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:36.197 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.197 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.197 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:36.197 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:36.197 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:36.197 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.197 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:36.197 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.197 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:36.197 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:36.197 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.197 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:36.198 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:36.198 INFO:tasks.workunit.client.0.smithi120.stderr:+++ umask 2024-02-06T21:42:36.198 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' 0022 -eq 0 ']' 2024-02-06T21:42:36.198 INFO:tasks.workunit.client.0.smithi120.stderr:++ SHELL=/bin/bash 2024-02-06T21:42:36.198 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.198 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-06T21:42:36.198 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.198 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-06T21:42:36.198 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-06T21:42:36.198 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:36.198 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:36.198 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:36.198 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.198 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.198 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.199 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-06T21:42:36.200 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.200 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:36.200 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.200 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:36.200 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:36.200 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.200 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:36.200 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.200 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:36.200 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:36.200 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:36.200 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:36.200 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41432/exe 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41522/exe 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'grep=grep --color=auto' 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.202 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorls.sh 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' '!' -t 0 ']' 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:+++ return 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export S_COLORS=auto 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:+++ S_COLORS=auto 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:36.203 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:36.204 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:36.204 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:36.204 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:36.204 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:36.204 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:36.204 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:36.204 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:36.204 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:36.204 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:36.204 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:36.204 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:36.204 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:36.204 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:36.204 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ read_operations: /' 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_neorados_read_operations.log 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ ec_list: /' 2024-02-06T21:42:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_neorados_read_operations 2024-02-06T21:42:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_neorados_ec_list.log 2024-02-06T21:42:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_neorados_ec_list 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41581/exe 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-06T21:42:36.207 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:36.208 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:36.208 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:36.208 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:36.208 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-06T21:42:36.208 INFO:tasks.workunit.client.0.smithi120.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-06T21:42:36.208 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.208 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-06T21:42:36.208 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.208 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-06T21:42:36.208 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-06T21:42:36.208 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.208 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:36.208 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-06T21:42:36.208 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.208 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/gawk.sh 2024-02-06T21:42:36.209 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:36.209 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.209 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:36.209 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-06T21:42:36.209 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.209 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/lang.sh 2024-02-06T21:42:36.209 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup 2024-02-06T21:42:36.209 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ watch_notify: /' 2024-02-06T21:42:36.209 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_neorados_watch_notify 2024-02-06T21:42:36.209 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.209 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:36.209 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/locale 2024-02-06T21:42:36.209 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.209 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:36.209 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:36.210 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-02-06T21:42:36.210 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:36.210 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:36.210 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:36.210 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:36.210 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.210 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-06T21:42:36.210 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.210 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.210 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:36.210 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.210 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:36.210 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:36.210 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41558/exe 2024-02-06T21:42:36.211 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:36.212 INFO:tasks.workunit.client.0.smithi120.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-06T21:42:36.212 INFO:tasks.workunit.client.0.smithi120.stderr:++++ export LANG=en_US.UTF-8 2024-02-06T21:42:36.212 INFO:tasks.workunit.client.0.smithi120.stderr:++++ LANG=en_US.UTF-8 2024-02-06T21:42:36.212 INFO:tasks.workunit.client.0.smithi120.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-06T21:42:36.212 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-06T21:42:36.212 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.212 INFO:tasks.workunit.client.0.smithi120.stderr:+++ LANG=en_US.UTF-8 2024-02-06T21:42:36.212 INFO:tasks.workunit.client.0.smithi120.stderr:+++ unset LANG_backup config 2024-02-06T21:42:36.212 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n '' ']' 2024-02-06T21:42:36.212 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-06T21:42:36.212 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' dumb = linux ']' 2024-02-06T21:42:36.212 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.212 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-06T21:42:36.212 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.212 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/less.sh 2024-02-06T21:42:36.213 INFO:tasks.workunit.client.0.smithi120.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-06T21:42:36.213 INFO:tasks.workunit.client.0.smithi120.stderr:++ for i in /etc/profile.d/*.sh 2024-02-06T21:42:36.213 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-06T21:42:36.213 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' '' ']' 2024-02-06T21:42:36.213 INFO:tasks.workunit.client.0.smithi120.stderr:++ . /etc/profile.d/which2.sh 2024-02-06T21:42:36.213 INFO:tasks.workunit.client.0.smithi120.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-06T21:42:36.213 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:36.213 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:36.213 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:36.213 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:36.213 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:36.213 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:36.213 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:36.213 INFO:tasks.workunit.client.0.smithi120.stderr:+++++ readlink /proc/41625/exe 2024-02-06T21:42:36.213 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_neorados_snapshots 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_neorados_snapshots.log 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ snapshots: /' 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:++++ basename /usr/bin/bash 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_declare='declare -f' 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:+++ which_opt=-f 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export which_declare 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:+++ export -f which 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset i 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:++ unset -f pathmunge 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:+ export PATH 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:+ unset rc 2024-02-06T21:42:36.214 INFO:tasks.workunit.client.0.smithi120.stderr:+ tee ceph_test_neorados_write_operations.log 2024-02-06T21:42:36.215 INFO:tasks.workunit.client.0.smithi120.stderr:+ ceph_test_neorados_write_operations 2024-02-06T21:42:36.215 INFO:tasks.workunit.client.0.smithi120.stderr:+ sed 's/^/ write_operations: /' 2024-02-06T21:42:38.635 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.717+0000 7fc1a9c35640 1 --2- 172.21.15.120:0/2499202281 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc19806b490 0x7fc19806d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:42:38.635 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.718+0000 7fc1ac43a640 1 --2- 172.21.15.120:0/2499202281 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc19806b490 0x7fc19806d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:42:38.635 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.718+0000 7fc1ac43a640 10 monclient: get_auth_request con 0x7fc19806b490 auth_method 0 2024-02-06T21:42:38.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.719+0000 7fc1a9c35640 10 client.4426.objecter ms_handle_connect 0x7fc19806b490 2024-02-06T21:42:38.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.719+0000 7fc1ac43a640 1 --2- 172.21.15.120:0/2499202281 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc19806b490 0x7fc19806d950 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00c010 tx=0x7fc19c002000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:42:38.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.720+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc1a0057310 con 0x55ef778e6b30 2024-02-06T21:42:38.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.720+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(23..23 src has 1..23) v4 2024-02-06T21:42:38.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.720+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [23,23] > 0 2024-02-06T21:42:38.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.720+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding full epoch 23 2024-02-06T21:42:38.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.720+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:42:38.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.720+0000 7fc1af40c780 10 librados: wait_for_osdmap done waiting 2024-02-06T21:42:38.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.720+0000 7fc1af40c780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:42:38.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.720+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55ef77858be0 con 0x55ef778e6b30 2024-02-06T21:42:38.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.733+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1a0005a80 con 0x55ef778e6b30 2024-02-06T21:42:38.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.733+0000 7fc1a9c35640 10 monclient: handle_get_version_reply finishing 1 version 23 2024-02-06T21:42:38.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.733+0000 7fc1aac37640 10 client.4426.objecter _get_latest_version latest 23, have it 2024-02-06T21:42:38.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.733+0000 7fc1af40c780 10 client.4426.objecter create_pool name=ceph_test_rados_api_asio 2024-02-06T21:42:38.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.733+0000 7fc1af40c780 10 client.4426.objecter pool_op_submit 1 2024-02-06T21:42:38.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.733+0000 7fc1af40c780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:42:38.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:35.733+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55ef7786a360 con 0x55ef778e6b30 2024-02-06T21:42:38.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.633+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:38.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.633+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:38.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.633+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:38.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.633+0000 7fc1a8c33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:42:38.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.820+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v25) v1 ==== 43+0+0 (secure 0 0 0) 0x7fc1a00570e0 con 0x55ef778e6b30 2024-02-06T21:42:38.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.820+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 pool_op_reply(tid 1 (0) Success v25) v1 2024-02-06T21:42:38.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.820+0000 7fc1a9c35640 10 client.4426.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v25) v1 2024-02-06T21:42:38.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.820+0000 7fc1a9c35640 10 client.4426.objecter have request 1 at 0x55ef7796d640 Op: create 2024-02-06T21:42:38.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.820+0000 7fc1a9c35640 20 client.4426.objecter waiting for client to reach epoch 25 before calling back 2024-02-06T21:42:38.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.820+0000 7fc1a9c35640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:42:38.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.820+0000 7fc1a9c35640 10 monclient: _renew_subs 2024-02-06T21:42:38.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.820+0000 7fc1a9c35640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.820+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=24}) v3 -- 0x7fc198071200 con 0x55ef778e6b30 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.820+0000 7fc1a9c35640 10 client.4426.objecter done 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 8 ==== osd_map(24..25 src has 1..25) v4 ==== 14876+0+0 (secure 0 0 0) 0x7fc1a005b3b0 con 0x55ef778e6b30 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(24..25 src has 1..25) v4 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [24,25] > 23 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 24 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 25 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1af40c780 10 librados: set snap read head -> 1 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1af40c780 10 librados: writefull oid=exist nspace= 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef778a6fc0 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 25 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1af40c780 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1af40c780 1 --2- 172.21.15.120:0/2499202281 >> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] conn(0x55ef778a7460 0x55ef77a297b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:42:38.906 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 2 osd.1 2024-02-06T21:42:38.907 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:42:45.687 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 2 2024-02-06T21:42:45.687 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1af40c780 15 client.4426.objecter _send_op 2 to 12.1 on osd.1 2024-02-06T21:42:45.687 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) v8 -- 0x55ef7796b3b0 con 0x55ef778a7460 2024-02-06T21:42:45.687 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:42:45.687 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:38.903+0000 7fc1af40c780 5 client.4426.objecter 1 in flight 2024-02-06T21:42:45.687 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:39.412+0000 7fc1acc3b640 1 --2- 172.21.15.120:0/2499202281 >> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] conn(0x55ef778a7460 0x55ef77a297b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T21:42:45.687 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:39.440+0000 7fc1acc3b640 10 monclient: get_auth_request con 0x55ef778a7460 auth_method 0 2024-02-06T21:42:45.687 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:39.519+0000 7fc1a9c35640 10 client.4426.objecter ms_handle_connect 0x55ef778a7460 2024-02-06T21:42:45.687 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:39.519+0000 7fc1acc3b640 1 --2- 172.21.15.120:0/2499202281 >> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] conn(0x55ef778a7460 0x55ef77a297b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T21:42:45.687 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:39.633+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:45.687 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:39.633+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:45.688 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:39.633+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:45.688 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:40.633+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:45.688 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:40.633+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:45.688 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:40.633+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:45.688 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:40.684+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:42:45.689 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:41.633+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:45.689 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:41.633+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:45.689 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:41.633+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:45.689 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:42.633+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:45.689 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:42.633+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:45.689 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:42.633+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:45.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:43.634+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:45.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:43.634+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:45.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:43.634+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:45.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:44.634+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:45.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:44.634+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:45.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:44.634+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:45.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:45.634+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:45.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:45.634+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:45.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:45.634+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:47.228 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-02-06T21:42:47.229 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: [----------] Global test environment set-up. 2024-02-06T21:42:47.229 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: [----------] 4 tests from LibRadosService 2024-02-06T21:42:47.229 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-02-06T21:42:47.229 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5299 ms) 2024-02-06T21:42:47.229 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-02-06T21:42:47.229 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: [ OK ] LibRadosService.RegisterLate (52 ms) 2024-02-06T21:42:47.229 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-02-06T21:42:47.229 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: cluster: 2024-02-06T21:42:47.229 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: id: 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:42:47.229 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: health: HEALTH_WARN 2024-02-06T21:42:47.229 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 34 pool(s) do not have an application enabled 2024-02-06T21:42:47.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 2024-02-06T21:42:47.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: services: 2024-02-06T21:42:47.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-02-06T21:42:47.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-02-06T21:42:47.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: osd: 8 osds: 8 up (since 73s), 8 in (since 2m) 2024-02-06T21:42:47.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-02-06T21:42:47.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 2024-02-06T21:42:47.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: data: 2024-02-06T21:42:47.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: pools: 37 pools, 1129 pgs 2024-02-06T21:42:47.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: objects: 2 objects, 449 KiB 2024-02-06T21:42:47.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-02-06T21:42:47.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: pgs: 87.245% pgs unknown 2024-02-06T21:42:47.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 11.780% pgs not active 2024-02-06T21:42:47.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 985 unknown 2024-02-06T21:42:47.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 120 creating+peering 2024-02-06T21:42:47.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 13 creating+activating 2024-02-06T21:42:47.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 11 active+clean 2024-02-06T21:42:47.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 2024-02-06T21:42:47.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 2024-02-06T21:42:47.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: cluster: 2024-02-06T21:42:47.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: id: 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:42:47.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: health: HEALTH_WARN 2024-02-06T21:42:47.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 17 pool(s) do not have an application enabled 2024-02-06T21:42:47.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 2024-02-06T21:42:47.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: services: 2024-02-06T21:42:47.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-02-06T21:42:47.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-02-06T21:42:47.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: osd: 8 osds: 8 up (since 75s), 8 in (since 2m) 2024-02-06T21:42:47.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 2024-02-06T21:42:47.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: data: 2024-02-06T21:42:47.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: pools: 37 pools, 1081 pgs 2024-02-06T21:42:47.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: objects: 2 objects, 449 KiB 2024-02-06T21:42:47.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-02-06T21:42:47.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: pgs: 74.931% pgs unknown 2024-02-06T21:42:47.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 24.237% pgs not active 2024-02-06T21:42:47.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 810 unknown 2024-02-06T21:42:47.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 262 creating+peering 2024-02-06T21:42:47.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 9 active+clean 2024-02-06T21:42:47.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 2024-02-06T21:42:47.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 2024-02-06T21:42:47.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: cluster: 2024-02-06T21:42:47.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: id: 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T21:42:47.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: health: HEALTH_WARN 2024-02-06T21:42:47.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 17 pool(s) do not have an application enabled 2024-02-06T21:42:47.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 2024-02-06T21:42:47.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: services: 2024-02-06T21:42:47.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-02-06T21:42:47.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-02-06T21:42:47.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: osd: 8 osds: 8 up (since 77s), 8 in (since 2m) 2024-02-06T21:42:47.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-02-06T21:42:47.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-02-06T21:42:47.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 2024-02-06T21:42:47.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: data: 2024-02-06T21:42:47.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: pools: 36 pools, 1049 pgs 2024-02-06T21:42:47.234 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: objects: 2 objects, 449 KiB 2024-02-06T21:42:47.234 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-02-06T21:42:47.234 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: pgs: 74.833% pgs unknown 2024-02-06T21:42:47.234 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 24.309% pgs not active 2024-02-06T21:42:50.598 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:02:30.412 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:50.603 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:02:30.418 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:50.674 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:30.438 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:50.719 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:02:30.531 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:50.762 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:02:30.576 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:50.785 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:30.549 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:50.859 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:02:30.674 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:50.941 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:45.685+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:42:50.941 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:46.634+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:50.941 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:46.634+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:50.941 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:46.634+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:50.941 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:47.634+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:50.941 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:47.634+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:50.941 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:47.634+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:50.941 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:48.634+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:50.941 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:48.634+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:50.941 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:48.634+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:50.941 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:48.634+0000 7fc1a8c33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:42:50.942 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:49.634+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:50.942 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:49.634+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:50.942 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:49.634+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:50.942 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.634+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:50.942 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.634+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:50.942 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.634+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:50.942 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.686+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:42:50.942 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.686+0000 7fc1ae69e640 2 client.4426.objecter tid 2 on osd.1 is laggy 2024-02-06T21:42:50.942 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.686+0000 7fc1ae69e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:42:50.942 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.686+0000 7fc1ae69e640 10 monclient: _renew_subs 2024-02-06T21:42:50.942 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.686+0000 7fc1ae69e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:42:50.943 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.686+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=26}) v3 -- 0x7fc184002f90 con 0x55ef778e6b30 2024-02-06T21:42:50.943 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.686+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- ping magic: 0 v1 -- 0x7fc184003320 con 0x55ef778a7460 2024-02-06T21:42:50.943 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 9 ==== osd_map(26..33 src has 1..33) v4 ==== 22011+0+0 (secure 0 0 0) 0x7fc1a005f970 con 0x55ef778e6b30 2024-02-06T21:42:50.943 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(26..33 src has 1..33) v4 2024-02-06T21:42:50.943 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [26,33] > 25 2024-02-06T21:42:50.943 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 26 2024-02-06T21:42:50.943 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:42:50.943 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 26 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:42:50.943 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:42:50.943 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 27 2024-02-06T21:42:50.964 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:30.728 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:50.996 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:02:30.811 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:42:51.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:42:51.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 27 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:42:51.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:42:51.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 28 2024-02-06T21:42:51.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:42:51.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 28 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:42:51.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:42:51.636 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 29 2024-02-06T21:42:51.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:42:51.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 29 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:42:51.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:42:51.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 30 2024-02-06T21:42:51.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:42:51.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 30 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:42:51.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:42:51.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 31 2024-02-06T21:42:51.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:42:51.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 31 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:42:51.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:42:51.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 32 2024-02-06T21:42:51.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:42:51.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 32 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:42:51.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:42:51.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 33 2024-02-06T21:42:51.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:42:51.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 33 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:42:51.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:42:51.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:42:51.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:50.939+0000 7fc1a9c35640 20 client.4426.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-02-06T21:42:51.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:51.634+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:55.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:51.634+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:55.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:51.634+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:55.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:52.634+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:55.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:52.634+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:55.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:52.634+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:55.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:53.635+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:55.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:53.635+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:55.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:53.635+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:55.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:54.635+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:55.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:54.635+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:55.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:54.635+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:55.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.635+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:42:55.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:42:55.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:42:55.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.687+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:42:55.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.687+0000 7fc1ae69e640 2 client.4426.objecter tid 2 on osd.1 is laggy 2024-02-06T21:42:55.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.687+0000 7fc1ae69e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:42:55.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.687+0000 7fc1ae69e640 10 monclient: _renew_subs 2024-02-06T21:42:55.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.687+0000 7fc1ae69e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:42:55.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.687+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=34}) v3 -- 0x7fc1840034e0 con 0x55ef778e6b30 2024-02-06T21:42:55.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.687+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- ping magic: 0 v1 -- 0x7fc184003730 con 0x55ef778a7460 2024-02-06T21:42:55.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 10 ==== osd_map(34..36 src has 1..36) v4 ==== 1939+0+0 (secure 0 0 0) 0x7fc1a005b120 con 0x55ef778e6b30 2024-02-06T21:42:55.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(34..36 src has 1..36) v4 2024-02-06T21:42:55.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [34,36] > 33 2024-02-06T21:42:55.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 34 2024-02-06T21:42:55.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:42:55.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 34 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:42:55.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:42:55.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 35 2024-02-06T21:42:55.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:42:55.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 35 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:00.692 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:00.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 36 2024-02-06T21:43:00.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:00.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 36 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:00.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:00.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:43:00.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:55.898+0000 7fc1a9c35640 20 client.4426.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-02-06T21:43:00.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:56.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:00.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:56.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:00.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:56.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:00.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:57.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:00.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:57.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:00.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:57.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:00.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:58.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:00.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:58.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:00.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:58.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:00.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:58.636+0000 7fc1a8c33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:43:00.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:59.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:00.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:59.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:00.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:42:59.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:00.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:00.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:00.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:00.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.688+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:43:00.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.688+0000 7fc1ae69e640 2 client.4426.objecter tid 2 on osd.1 is laggy 2024-02-06T21:43:00.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.688+0000 7fc1ae69e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:43:00.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.688+0000 7fc1ae69e640 10 monclient: _renew_subs 2024-02-06T21:43:00.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.688+0000 7fc1ae69e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:43:00.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.688+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=37}) v3 -- 0x7fc1840038c0 con 0x55ef778e6b30 2024-02-06T21:43:00.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.688+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- ping magic: 0 v1 -- 0x7fc184003b60 con 0x55ef778a7460 2024-02-06T21:43:00.696 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 11 ==== osd_map(37..39 src has 1..39) v4 ==== 1915+0+0 (secure 0 0 0) 0x7fc1a0057080 con 0x55ef778e6b30 2024-02-06T21:43:00.696 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(37..39 src has 1..39) v4 2024-02-06T21:43:02.528 INFO:tasks.workunit.client.0.smithi120.stdout: pool: Running main() from gmock_main.cc 2024-02-06T21:43:02.528 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-02-06T21:43:02.528 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [----------] Global test environment set-up. 2024-02-06T21:43:02.529 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-02-06T21:43:02.529 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-02-06T21:43:02.529 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [ OK ] NeoRadosPools.PoolList (6096 ms) 2024-02-06T21:43:02.529 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-02-06T21:43:02.529 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (3927 ms) 2024-02-06T21:43:02.529 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-02-06T21:43:02.529 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (3342 ms) 2024-02-06T21:43:02.529 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-02-06T21:43:02.529 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (7815 ms) 2024-02-06T21:43:02.529 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-02-06T21:43:02.529 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2674 ms) 2024-02-06T21:43:02.529 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-02-06T21:43:02.529 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2680 ms) 2024-02-06T21:43:02.529 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [----------] 6 tests from NeoRadosPools (26534 ms total) 2024-02-06T21:43:02.530 INFO:tasks.workunit.client.0.smithi120.stdout: pool: 2024-02-06T21:43:02.530 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [----------] Global test environment tear-down 2024-02-06T21:43:02.530 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [==========] 6 tests from 1 test suite ran. (26534 ms total) 2024-02-06T21:43:02.530 INFO:tasks.workunit.client.0.smithi120.stdout: pool: [ PASSED ] 6 tests. 2024-02-06T21:43:05.689 INFO:tasks.workunit.client.0.smithi120.stdout:api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [37,39] > 36 2024-02-06T21:43:05.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 37 2024-02-06T21:43:05.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:05.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 37 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:05.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:05.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 38 2024-02-06T21:43:05.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:05.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 38 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:05.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:05.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 39 2024-02-06T21:43:05.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:05.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 39 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:05.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:05.690 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:00.692+0000 7fc1a9c35640 20 client.4426.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:01.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:01.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:01.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:02.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:02.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:02.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:03.635+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:03.635+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:03.635+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:04.635+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:04.635+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:04.635+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.635+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.635+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:05.691 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.635+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:05.692 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.688+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:43:05.692 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.688+0000 7fc1ae69e640 2 client.4426.objecter tid 2 on osd.1 is laggy 2024-02-06T21:43:05.692 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.688+0000 7fc1ae69e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:43:07.338 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.688+0000 7fc1ae69e640 10 monclient: _renew api_service: 785 unknown 2024-02-06T21:43:07.338 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 255 creating+peering 2024-02-06T21:43:07.338 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 9 active+clean 2024-02-06T21:43:07.338 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 2024-02-06T21:43:07.338 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 2024-02-06T21:43:07.338 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: [ OK ] LibRadosService.StatusFormat (6125 ms) 2024-02-06T21:43:07.338 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: [ RUN ] LibRadosService.Status 2024-02-06T21:43:07.338 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: [ OK ] LibRadosService.Status (20069 ms) 2024-02-06T21:43:07.338 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: [----------] 4 tests from LibRadosService (31545 ms total) 2024-02-06T21:43:07.338 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: 2024-02-06T21:43:07.338 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: [----------] Global test environment tear-down 2024-02-06T21:43:07.338 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31545 ms total) 2024-02-06T21:43:07.338 INFO:tasks.workunit.client.0.smithi120.stdout: api_service: [ PASSED ] 4 tests. 2024-02-06T21:43:07.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-02-06T21:43:07.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: [----------] Global test environment set-up. 2024-02-06T21:43:07.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-02-06T21:43:07.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-02-06T21:43:07.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5672 ms) 2024-02-06T21:43:07.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-02-06T21:43:07.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (674 ms) 2024-02-06T21:43:07.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-02-06T21:43:07.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20349 ms) 2024-02-06T21:43:07.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-02-06T21:43:07.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: attempt 0 of 20 2024-02-06T21:43:07.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: failed to deregister: 2024-02-06T21:43:07.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: { 2024-02-06T21:43:07.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "epoch": 11, 2024-02-06T21:43:07.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "modified": "2024-02-06T21:43:03.595051+0000", 2024-02-06T21:43:07.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "services": { 2024-02-06T21:43:07.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "laundry": { 2024-02-06T21:43:07.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "daemons": { 2024-02-06T21:43:07.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "summary": "", 2024-02-06T21:43:07.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "close-test-pid41097": { 2024-02-06T21:43:07.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "start_epoch": 11, 2024-02-06T21:43:07.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "start_stamp": "2024-02-06T21:43:02.592636+0000", 2024-02-06T21:43:07.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "gid": 4640, 2024-02-06T21:43:07.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "addr": "172.21.15.120:0/661167121", 2024-02-06T21:43:07.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "metadata": { 2024-02-06T21:43:07.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "arch": "x86_64", 2024-02-06T21:43:07.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "ceph_release": "squid", 2024-02-06T21:43:07.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1269-g633ab857 (633ab857b9926af935a3e6291c3e1d9251aca357) squid (dev)", 2024-02-06T21:43:07.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "ceph_version_short": "19.0.0-1269-g633ab857", 2024-02-06T21:43:07.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-02-06T21:43:07.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "distro": "centos", 2024-02-06T21:43:07.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-02-06T21:43:07.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "distro_version": "9", 2024-02-06T21:43:07.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "foo": "bar", 2024-02-06T21:43:07.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "hostname": "smithi120", 2024-02-06T21:43:07.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Feb 1 21:06:52 UTC 2024", 2024-02-06T21:43:07.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "kernel_version": "5.14.0-417.el9.x86_64", 2024-02-06T21:43:07.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "mem_swap_kb": "0", 2024-02-06T21:43:07.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "mem_total_kb": "32488404", 2024-02-06T21:43:07.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "os": "Linux", 2024-02-06T21:43:07.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "this": "that" 2024-02-06T21:43:07.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: }, 2024-02-06T21:43:07.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "task_status": {} 2024-02-06T21:43:07.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: }, 2024-02-06T21:43:07.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "pid41033": { 2024-02-06T21:43:07.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "start_epoch": 10, 2024-02-06T21:43:07.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "start_stamp": "2024-02-06T21:42:47.347763+0000", 2024-02-06T21:43:07.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "gid": 4607, 2024-02-06T21:43:07.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "addr": "172.21.15.120:0/1664244026", 2024-02-06T21:43:07.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "metadata": { 2024-02-06T21:43:07.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "arch": "x86_64", 2024-02-06T21:43:07.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "ceph_release": "squid", 2024-02-06T21:43:07.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1269-g633ab857 (633ab857b9926af935a3e6291c3e1d9251aca357) squid (dev)", 2024-02-06T21:43:07.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "ceph_version_short": "19.0.0-1269-g633ab857", 2024-02-06T21:43:07.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-02-06T21:43:08.637 INFO:tasks.workunit.client.0.smithi120.stdout: _subs 2024-02-06T21:43:08.637 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.688+0000 7fc1ae69e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:43:08.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.688+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=40}) v3 -- 0x7fc1840038c0 con 0x55ef778e6b30 2024-02-06T21:43:08.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.688+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- ping magic: 0 v1 -- 0x7fc184003cf0 con 0x55ef778a7460 2024-02-06T21:43:08.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.691+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 12 ==== osd_map(40..42 src has 1..42) v4 ==== 2636+0+0 (secure 0 0 0) 0x7fc1a0053980 con 0x55ef778e6b30 2024-02-06T21:43:08.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.691+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(40..42 src has 1..42) v4 2024-02-06T21:43:08.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.691+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [40,42] > 39 2024-02-06T21:43:08.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.691+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 40 2024-02-06T21:43:08.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.691+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:08.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.691+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 40 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:08.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.691+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:08.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.691+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 41 2024-02-06T21:43:08.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.691+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:08.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.691+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 41 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:08.638 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.691+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:08.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.691+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 42 2024-02-06T21:43:08.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.692+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:08.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.692+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 42 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:08.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.692+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:08.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.692+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:43:08.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:05.692+0000 7fc1a9c35640 20 client.4426.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-02-06T21:43:08.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:06.635+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:08.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:06.635+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:08.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:06.635+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:08.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:07.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:08.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:07.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:08.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:07.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:08.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:08.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:08.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:08.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:08.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:08.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:09.092 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:08.636+0000 7fc1a8c33640 20 monclient: _un_ba handler_error: Running main() from gmock_main.cc 2024-02-06T21:43:09.093 INFO:tasks.workunit.client.0.smithi120.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-02-06T21:43:09.093 INFO:tasks.workunit.client.0.smithi120.stdout: handler_error: [----------] Global test environment set-up. 2024-02-06T21:43:09.093 INFO:tasks.workunit.client.0.smithi120.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-02-06T21:43:09.093 INFO:tasks.workunit.client.0.smithi120.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-02-06T21:43:09.093 INFO:tasks.workunit.client.0.smithi120.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (33136 ms) 2024-02-06T21:43:09.093 INFO:tasks.workunit.client.0.smithi120.stdout: handler_error: [----------] 1 test from neocls_handler_error (33136 ms total) 2024-02-06T21:43:09.093 INFO:tasks.workunit.client.0.smithi120.stdout: handler_error: 2024-02-06T21:43:09.093 INFO:tasks.workunit.client.0.smithi120.stdout: handler_error: [----------] Global test environment tear-down 2024-02-06T21:43:09.093 INFO:tasks.workunit.client.0.smithi120.stdout: handler_error: [==========] 1 test from 1 test suite ran. (33136 ms total) 2024-02-06T21:43:09.093 INFO:tasks.workunit.client.0.smithi120.stdout: handler_error: [ PASSED ] 1 test. 2024-02-06T21:43:09.687 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:09.676+0000 12a5a640 -1 osd.1 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4494.0:2 10.1a 10:5c6776db:::0.obj:head [write 0~31 in=31b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:10.056 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:09.907+0000 12a5a640 -1 osd.2 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:2 7.1 7:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:10.694 INFO:tasks.workunit.client.0.smithi120.stdout:ckoff reopen_interval_multipler now 1 2024-02-06T21:43:10.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:09.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:10.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:09.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:10.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:09.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:10.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:10.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:10.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:10.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.689+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:43:10.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.689+0000 7fc1ae69e640 2 client.4426.objecter tid 2 on osd.1 is laggy 2024-02-06T21:43:10.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.689+0000 7fc1ae69e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:43:10.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.689+0000 7fc1ae69e640 10 monclient: _renew_subs 2024-02-06T21:43:10.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.689+0000 7fc1ae69e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:43:10.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.689+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=43}) v3 -- 0x7fc1840038c0 con 0x55ef778e6b30 2024-02-06T21:43:10.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.689+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- ping magic: 0 v1 -- 0x7fc184003ee0 con 0x55ef778a7460 2024-02-06T21:43:10.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 13 ==== osd_map(43..45 src has 1..45) v4 ==== 3395+0+0 (secure 0 0 0) 0x7fc1a005d370 con 0x55ef778e6b30 2024-02-06T21:43:10.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(43..45 src has 1..45) v4 2024-02-06T21:43:10.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [43,45] > 42 2024-02-06T21:43:10.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 43 2024-02-06T21:43:10.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:10.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 43 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:10.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:10.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 44 2024-02-06T21:43:10.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:10.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 44 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:10.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:10.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 45 2024-02-06T21:43:10.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:10.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 45 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:10.695 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:10.696 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:43:10.709 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:10.671+0000 12a5a640 -1 osd.1 43 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4494.0:2 10.1a 10:5c6776db:::0.obj:head [write 0~31 in=31b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:10.968 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:43:10.904+0000 12a5a640 -1 osd.5 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:2 25.6 25:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:11.108 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:11.104+0000 12a5a640 -1 osd.2 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4373.0:2 4.1 4.dc849861 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:11.490 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:11.480+0000 ea52640 -1 osd.0 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35.7c7d9ba6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:11.721 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:11.690+0000 12a5a640 -1 osd.3 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4382.0:2 18.19 18:9b2afc17:test-rados-api-smithi120-40610-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:11.966 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:43:11.947+0000 12a5a640 -1 osd.5 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:2 25.6 25:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:12.096 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:12.086+0000 12a5a640 -1 osd.1 44 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:2 12.1 12.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:12.163 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:12.037+0000 12a5a640 -1 osd.6 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17.f3ad50be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:12.805 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:12.775+0000 ea52640 -1 osd.0 45 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35.7c7d9ba6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:12.864 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:12.842+0000 12a5a640 -1 osd.3 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4382.0:2 18.19 18:9b2afc17:test-rados-api-smithi120-40610-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:13.071 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:43:12.988+0000 12a5a640 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:2 25.6 25:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:13.135 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:13.131+0000 12a5a640 -1 osd.1 45 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:2 12.1 12.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:13.396 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:10.692+0000 7fc api_service_pp: "distro": "centos", 2024-02-06T21:43:13.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-02-06T21:43:13.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "distro_version": "9", 2024-02-06T21:43:13.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "foo": "bar", 2024-02-06T21:43:13.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "hostname": "smithi120", 2024-02-06T21:43:13.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Feb 1 21:06:52 UTC 2024", 2024-02-06T21:43:13.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "kernel_version": "5.14.0-417.el9.x86_64", 2024-02-06T21:43:13.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "mem_swap_kb": "0", 2024-02-06T21:43:13.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "mem_total_kb": "32488404", 2024-02-06T21:43:13.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "os": "Linux", 2024-02-06T21:43:13.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "this": "that" 2024-02-06T21:43:13.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: }, 2024-02-06T21:43:13.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: "task_status": {} 2024-02-06T21:43:13.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: } 2024-02-06T21:43:13.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: } 2024-02-06T21:43:13.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: } 2024-02-06T21:43:13.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: } 2024-02-06T21:43:13.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: } 2024-02-06T21:43:13.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: 2024-02-06T21:43:13.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: attempt 1 of 20 2024-02-06T21:43:13.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10904 ms) 2024-02-06T21:43:13.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (37599 ms total) 2024-02-06T21:43:13.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: 2024-02-06T21:43:13.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: [----------] Global test environment tear-down 2024-02-06T21:43:13.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (37599 ms total) 2024-02-06T21:43:13.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-02-06T21:43:13.901 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:13.835+0000 12a5a640 -1 osd.3 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4382.0:2 18.19 18:9b2afc17:test-rados-api-smithi120-40610-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:14.040 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:13.933+0000 12a5a640 -1 osd.2 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4373.0:2 4.1 4.dc849861 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:14.130 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:43:14.106+0000 12a5a640 -1 osd.5 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:2 25.6 25:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:14.183 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:14.121+0000 12a5a640 -1 osd.6 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17.f3ad50be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:14.415 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:14.335+0000 12a5a640 -1 osd.1 46 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:2 12.1 12.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:14.543 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:14.537+0000 ea52640 -1 osd.0 46 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35.7c7d9ba6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:14.986 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:14.977+0000 12a5a640 -1 osd.2 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4373.0:2 4.1 4:8619213b:test-rados-api-smithi120-40537-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:15.215 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:02:55.030 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:15.219 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:02:55.034 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:15.226 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:02:55.040 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:15.228 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:15.194+0000 12a5a640 -1 osd.3 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4382.0:2 18.19 18:9b2afc17:test-rados-api-smithi120-40610-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:15.248 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:43:15.185+0000 12a5a640 -1 osd.5 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:2 25.6 25:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:15.274 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:15.262+0000 12a5a640 -1 osd.6 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17.f3ad50be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:15.279 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:55.043 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:15.282 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:02:55.097 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:15.284 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:55.048 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:15.428 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:15.397+0000 12a5a640 -1 osd.1 46 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:2 12.1 12.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:15.468 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:02:55.283 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:15.607 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:02:55.422 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:15.755 INFO:tasks.workunit.client.0.smithi120.stdout:1a9c35640 20 client.4426.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-02-06T21:43:15.755 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:11.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:15.755 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:11.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:15.755 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:11.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:15.756 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:12.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:15.756 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:12.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:15.756 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:12.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:15.756 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:13.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:15.756 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:13.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:15.756 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:13.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:15.756 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:14.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:15.756 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:14.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:15.756 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:14.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:15.756 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:15.756 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:15.756 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:15.756 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.690+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:43:15.756 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.690+0000 7fc1ae69e640 2 client.4426.objecter tid 2 on osd.1 is laggy 2024-02-06T21:43:15.757 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.690+0000 7fc1ae69e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:43:15.757 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.690+0000 7fc1ae69e640 10 monclient: _renew_subs 2024-02-06T21:43:15.757 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.690+0000 7fc1ae69e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:43:15.757 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.690+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=46}) v3 -- 0x7fc1840038c0 con 0x55ef778e6b30 2024-02-06T21:43:15.757 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.690+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- ping magic: 0 v1 -- 0x7fc184004120 con 0x55ef778a7460 2024-02-06T21:43:15.757 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.753+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 14 ==== osd_map(46..49 src has 1..49) v4 ==== 2211+0+0 (secure 0 0 0) 0x7fc1a005d530 con 0x55ef778e6b30 2024-02-06T21:43:15.757 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.753+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(46..49 src has 1..49) v4 2024-02-06T21:43:15.757 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.753+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [46,49] > 45 2024-02-06T21:43:15.757 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.753+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 46 2024-02-06T21:43:15.757 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.753+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:15.757 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 46 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:15.757 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:15.757 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 47 2024-02-06T21:43:15.757 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:15.761 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:55.525 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:16.183 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:16.175+0000 ea52640 -1 osd.0 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35.7c7d9ba6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:16.388 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:16.327+0000 12a5a640 -1 osd.6 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17.f3ad50be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:16.516 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:16.375+0000 12a5a640 -1 osd.2 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4373.0:2 4.1 4:8619213b:test-rados-api-smithi120-40537-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:16.542 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:16.348+0000 12a5a640 -1 osd.3 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4382.0:2 18.19 18:9b2afc17:test-rados-api-smithi120-40610-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:16.609 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:43:16.528+0000 12a5a640 -1 osd.5 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:2 25.6 25:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:16.959 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:16.781+0000 12a5a640 -1 osd.1 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4426.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:17.188 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:17.178+0000 ea52640 -1 osd.0 49 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35.7c7d9ba6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:17.438 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:17.428+0000 12a5a640 -1 osd.3 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4382.0:2 18.19 18:9b2afc17:test-rados-api-smithi120-40610-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:17.477 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:17.360+0000 12a5a640 -1 osd.6 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17.f3ad50be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:17.591 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:43:17.584+0000 12a5a640 -1 osd.5 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:2 25.6 25:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:17.716 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:17.671+0000 12a5a640 -1 osd.2 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4373.0:2 4.1 4:8619213b:test-rados-api-smithi120-40537-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:17.832 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:17.822+0000 12a5a640 -1 osd.1 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4426.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:18.257 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:18.248+0000 ea52640 -1 osd.0 50 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35.7c7d9ba6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:18.562 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:18.447+0000 12a5a640 -1 osd.6 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17.f3ad50be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:18.585 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:18.571+0000 12a5a640 -1 osd.3 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4382.0:2 18.19 18:9b2afc17:test-rados-api-smithi120-40610-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:18.653 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:43:18.641+0000 12a5a640 -1 osd.5 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:2 25.6 25:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:18.858 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:18.843+0000 12a5a640 -1 osd.1 49 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4426.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:19.484 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:19.308+0000 12a5a640 -1 osd.2 49 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4373.0:2 4.1 4:8619213b:test-rados-api-smithi120-40537-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:19.521 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:19.500+0000 ea52640 -1 osd.0 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35.7c7d9ba6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:19.834 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:19.693+0000 12a5a640 -1 osd.3 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4382.0:2 18.19 18:9b2afc17:test-rados-api-smithi120-40610-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:19.883 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:43:19.872+0000 12a5a640 -1 osd.5 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:2 25.6 25:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:19.908 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:19.819+0000 12a5a640 -1 osd.6 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17.f3ad50be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:20.074 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:20.066+0000 12a5a640 -1 osd.1 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4426.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:20.440 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:20.378+0000 12a5a640 -1 osd.2 50 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4373.0:2 4.1 4:8619213b:test-rados-api-smithi120-40537-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:20.564 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:20.541+0000 ea52640 -1 osd.0 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35.7c7d9ba6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:20.692 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 47 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:20.692 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:20.692 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 48 2024-02-06T21:43:20.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:20.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 48 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:20.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:20.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 49 2024-02-06T21:43:20.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:20.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 49 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:20.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:20.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:43:20.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:15.754+0000 7fc1a9c35640 20 client.4426.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-02-06T21:43:20.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:16.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:20.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:16.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:20.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:16.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:20.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:17.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:20.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:17.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:20.693 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:17.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:20.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:18.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:20.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:18.636+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:20.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:18.636+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:20.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:18.636+0000 7fc1a8c33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:43:20.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:19.636+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:20.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:19.637+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:20.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:19.637+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:20.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.637+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:20.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.637+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:20.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.637+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:20.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.691+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:43:20.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.691+0000 7fc1ae69e640 2 client.4426.objecter tid 2 on osd.1 is laggy 2024-02-06T21:43:20.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.691+0000 7fc1ae69e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:43:20.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.691+0000 7fc1ae69e640 10 monclient: _renew_subs 2024-02-06T21:43:20.694 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.691+0000 7fc1ae69e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:43:20.734 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:20.725+0000 12a5a640 -1 osd.3 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4382.0:2 18.19 18:9b2afc17:test-rados-api-smithi120-40610-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:21.340 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T21:43:21.309+0000 12a5a640 -1 osd.5 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:2 25.6 25:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:21.348 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:21.334+0000 12a5a640 -1 osd.6 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17.f3ad50be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:21.427 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:21.324+0000 12a5a640 -1 osd.1 51 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4426.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:21.973 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:21.894+0000 12a5a640 -1 osd.2 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4373.0:2 4.1 4:8619213b:test-rados-api-smithi120-40537-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:22.375 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:22.161+0000 12a5a640 -1 osd.3 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 13.6 13.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:22.530 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:22.469+0000 ea52640 -1 osd.0 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35.7c7d9ba6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:22.606 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:22.491+0000 12a5a640 -1 osd.1 53 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4426.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2024-02-06T21:43:22.893 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:22.884+0000 12a5a640 -1 osd.6 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17.f3ad50be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:23.238 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:23.154+0000 12a5a640 -1 osd.2 52 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4373.0:2 4.1 4:8619213b:test-rados-api-smithi120-40537-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:23.556 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:23.548+0000 12a5a640 -1 osd.1 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:23.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.691+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=50}) v3 -- 0x7fc1840038c0 con 0x55ef778e6b30 2024-02-06T21:43:23.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.691+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- ping magic: 0 v1 -- 0x7fc1840042b0 con 0x55ef778a7460 2024-02-06T21:43:23.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 15 ==== osd_map(50..53 src has 1..53) v4 ==== 2979+0+0 (secure 0 0 0) 0x7fc1a005d530 con 0x55ef778e6b30 2024-02-06T21:43:23.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(50..53 src has 1..53) v4 2024-02-06T21:43:23.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [50,53] > 49 2024-02-06T21:43:23.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 50 2024-02-06T21:43:23.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:23.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 50 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:23.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:23.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 51 2024-02-06T21:43:23.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:23.639 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 51 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:23.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:23.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 52 2024-02-06T21:43:23.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:23.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 52 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:23.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:23.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 53 2024-02-06T21:43:23.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:23.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 53 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:23.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:23.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:43:23.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:20.694+0000 7fc1a9c35640 20 client.4426.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-02-06T21:43:23.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:21.637+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:23.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:21.637+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:23.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:21.637+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:23.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:22.637+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:23.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:22.637+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:23.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:22.637+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:23.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:23.637+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:23.832 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:23.748+0000 ea52640 -1 osd.0 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35.7c7d9ba6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:24.112 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:24.098+0000 12a5a640 -1 osd.3 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 13.6 13.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:24.161 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:24.151+0000 12a5a640 -1 osd.6 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17.f3ad50be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:24.492 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:24.484+0000 12a5a640 -1 osd.2 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4373.0:2 4.1 4:8619213b:test-rados-api-smithi120-40537-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:24.633 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:24.619+0000 12a5a640 -1 osd.1 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:24.929 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:24.915+0000 ea52640 -1 osd.0 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35.7c7d9ba6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:25.474 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:25.372+0000 12a5a640 -1 osd.3 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 13.6 13.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:25.544 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:25.536+0000 12a5a640 -1 osd.2 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:25.585 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:25.509+0000 12a5a640 -1 osd.6 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17.f3ad50be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:25.863 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:25.781+0000 12a5a640 -1 osd.1 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:26.223 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:23.637+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:26.223 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:23.637+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:26.223 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:24.637+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:26.223 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:24.637+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:26.223 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:24.637+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:26.223 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:25.637+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:26.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:25.637+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:26.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:25.637+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:26.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:25.692+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:43:26.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:25.692+0000 7fc1ae69e640 2 client.4426.objecter tid 2 on osd.1 is laggy 2024-02-06T21:43:26.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:25.692+0000 7fc1ae69e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:43:26.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:25.692+0000 7fc1ae69e640 10 monclient: _renew_subs 2024-02-06T21:43:26.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:25.692+0000 7fc1ae69e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:43:26.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:25.692+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=54}) v3 -- 0x7fc1840038c0 con 0x55ef778e6b30 2024-02-06T21:43:26.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:25.692+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- ping magic: 0 v1 -- 0x7fc1840042b0 con 0x55ef778a7460 2024-02-06T21:43:26.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 16 ==== osd_map(54..56 src has 1..56) v4 ==== 1906+0+0 (secure 0 0 0) 0x7fc1a005d530 con 0x55ef778e6b30 2024-02-06T21:43:26.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(54..56 src has 1..56) v4 2024-02-06T21:43:26.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [54,56] > 53 2024-02-06T21:43:26.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 54 2024-02-06T21:43:26.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:26.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 54 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:26.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:26.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 55 2024-02-06T21:43:26.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:26.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 55 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:26.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:26.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 56 2024-02-06T21:43:26.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 10 client.4426.objecter checking op 2 2024-02-06T21:43:26.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 56 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:43:26.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:26.566 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:26.556+0000 12a5a640 -1 osd.2 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:26.606 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:26.544+0000 12a5a640 -1 osd.3 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 13.6 13.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:26.771 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:26.718+0000 ea52640 -1 osd.0 56 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35.7c7d9ba6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:26.773 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:26.768+0000 12a5a640 -1 osd.1 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:27.069 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:27.000+0000 12a5a640 -1 osd.6 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17.f3ad50be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:27.721 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:27.712+0000 12a5a640 -1 osd.3 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 13.6 13.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:27.798 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:27.740+0000 12a5a640 -1 osd.2 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:27.838 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:27.825+0000 12a5a640 -1 osd.1 56 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:28.106 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:28.100+0000 12a5a640 -1 osd.6 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17.f3ad50be (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:28.441 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:28.337+0000 ea52640 -1 osd.0 57 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35.7c7d9ba6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:28.861 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:28.854+0000 12a5a640 -1 osd.1 57 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:28.910 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:28.872+0000 12a5a640 -1 osd.3 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 13.6 13.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:29.331 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:29.192+0000 12a5a640 -1 osd.2 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:29.426 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:29.416+0000 ea52640 -1 osd.0 57 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35.7c7d9ba6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:29.511 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:29.443+0000 12a5a640 -1 osd.6 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:29.828 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:29.823+0000 12a5a640 -1 osd.1 57 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:29.840 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:43:29.840 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.221+0000 7fc1a9c35640 20 client.4426.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-02-06T21:43:29.841 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.637+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:29.841 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.637+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:29.841 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:26.637+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:29.841 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:27.637+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:29.841 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:27.637+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:29.841 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:27.637+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:29.841 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:28.637+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:29.841 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:28.637+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:29.841 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:28.637+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:29.841 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:28.637+0000 7fc1a8c33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:43:29.841 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.637+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:29.841 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.637+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:29.841 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.637+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:29.842 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.838+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 1 ==== osd_map(26..46 src has 1..46) v4 ==== 32052+0+0 (crc 0 0 0) 0x7fc1a4062040 con 0x55ef778a7460 2024-02-06T21:43:29.842 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.838+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(26..46 src has 1..46) v4 2024-02-06T21:43:29.842 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.838+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map ignoring epochs [26,46] <= 56 2024-02-06T21:43:29.842 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.838+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:43:29.842 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.838+0000 7fc1a9c35640 20 client.4426.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-02-06T21:43:29.842 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 2 ==== osd_op_reply(2 exist [writefull 0~5] v46'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:43:29.842 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(2 exist [writefull 0~5] v46'1 uv1 ondisk = 0) v8 2024-02-06T21:43:29.842 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:43:29.842 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 2 ondisk uv 1 in 12.1 attempt 0 2024-02-06T21:43:29.842 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 0 2024-02-06T21:43:29.842 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 2 2024-02-06T21:43:29.842 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 2 2024-02-06T21:43:29.842 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:43:29.842 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 2 2024-02-06T21:43:29.842 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1acc3b640 5 client.4426.objecter 0 in flight 2024-02-06T21:43:29.843 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 10 librados: Objecter returned from writefull r=0 2024-02-06T21:43:29.843 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-02-06T21:43:30.291 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:30.231+0000 12a5a640 -1 osd.3 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 13.6 13.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:30.381 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:30.317+0000 12a5a640 -1 osd.2 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:30.597 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:30.503+0000 ea52640 -1 osd.0 58 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:30.883 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef778a6fc0 2024-02-06T21:43:30.883 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 56 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:30.883 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:30.883 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:43:30.883 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:43:30.883 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid exist '@12' '@12' [read 0~256] tid 3 osd.1 2024-02-06T21:43:30.884 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:43:30.884 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 3 2024-02-06T21:43:30.884 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 15 client.4426.objecter _send_op 3 to 12.1 on osd.1 2024-02-06T21:43:30.884 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:3 12.1 12:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v8 -- 0x55ef7796b830 con 0x55ef778a7460 2024-02-06T21:43:30.884 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:43:30.884 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 5 client.4426.objecter 1 in flight 2024-02-06T21:43:30.884 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef7796bbe0 2024-02-06T21:43:30.884 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 56 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:30.884 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:30.884 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:43:30.884 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 4 2024-02-06T21:43:30.885 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid noexist '@12' '@12' [read 0~256] tid 4 osd.1 2024-02-06T21:43:30.885 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 4 2024-02-06T21:43:30.885 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 4 2024-02-06T21:43:30.885 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 15 client.4426.objecter _send_op 4 to 12.1 on osd.1 2024-02-06T21:43:30.885 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:4 12.1 12:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v8 -- 0x55ef7796c080 con 0x55ef778a7460 2024-02-06T21:43:30.885 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 5 2024-02-06T21:43:30.885 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:29.839+0000 7fc1af40c780 5 client.4426.objecter 2 in flight 2024-02-06T21:43:30.885 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.637+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:30.885 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.637+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:30.885 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.637+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:30.885 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.693+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:43:30.885 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 3 ==== osd_map(57..57 src has 1..57) v4 ==== 320+0+0 (crc 0 0 0) 0x7fc1a4062040 con 0x55ef778a7460 2024-02-06T21:43:30.885 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 4 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:43:30.885 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(57..57 src has 1..57) v4 2024-02-06T21:43:30.886 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-02-06T21:43:30.886 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:43:30.886 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [57,57] > 56 2024-02-06T21:43:30.886 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 57 2024-02-06T21:43:30.886 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1a9c35640 10 client.4426.objecter checking op 3 2024-02-06T21:43:30.886 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 57 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:30.886 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:30.886 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1a9c35640 10 client.4426.objecter checking op 4 2024-02-06T21:43:30.886 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 57 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:30.886 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:30.886 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:43:30.886 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1a9c35640 20 client.4426.objecter 3 12.d3a59201 osd.1 exist [read 0~256] 2024-02-06T21:43:30.886 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1a9c35640 20 client.4426.objecter 4 12.a7bbaa1 osd.1 noexist [read 0~256] 2024-02-06T21:43:30.886 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 3 ondisk uv 1 in 12.1 attempt 0 2024-02-06T21:43:30.887 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 5 2024-02-06T21:43:30.887 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 3 2024-02-06T21:43:30.887 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 3 2024-02-06T21:43:30.887 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 5 2024-02-06T21:43:30.887 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 3 2024-02-06T21:43:30.887 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 5 client.4426.objecter 1 in flight 2024-02-06T21:43:30.887 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 5 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:43:30.887 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-02-06T21:43:30.887 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:43:30.887 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 4 ondisk uv 0 in 12.1 attempt 0 2024-02-06T21:43:30.887 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 10 client.4426.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-02-06T21:43:30.887 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 0 2024-02-06T21:43:30.887 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 4 2024-02-06T21:43:30.888 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 4 2024-02-06T21:43:30.888 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:43:30.888 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 4 2024-02-06T21:43:30.888 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1acc3b640 5 client.4426.objecter 0 in flight 2024-02-06T21:43:30.888 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1042 ms) 2024-02-06T21:43:30.888 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-02-06T21:43:30.888 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef778a6fc0 2024-02-06T21:43:30.888 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 57 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:30.888 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:30.888 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:43:30.888 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:43:30.888 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid exist '@12' '@12' [read 0~256] tid 5 osd.1 2024-02-06T21:43:30.888 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:43:30.889 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 5 2024-02-06T21:43:30.889 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 15 client.4426.objecter _send_op 5 to 12.1 on osd.1 2024-02-06T21:43:30.889 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:5 12.1 12:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57) v8 -- 0x55ef7796c460 con 0x55ef778a7460 2024-02-06T21:43:30.889 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:43:30.889 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 5 client.4426.objecter 1 in flight 2024-02-06T21:43:30.889 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef7796bb90 2024-02-06T21:43:30.889 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 57 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:30.889 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:30.889 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:43:30.889 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 4 2024-02-06T21:43:30.889 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid noexist '@12' '@12' [read 0~256] tid 6 osd.1 2024-02-06T21:43:30.889 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 4 2024-02-06T21:43:30.889 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 6 2024-02-06T21:43:30.889 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 15 client.4426.objecter _send_op 6 to 12.1 on osd.1 2024-02-06T21:43:30.911 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:30.810+0000 12a5a640 -1 osd.6 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:31.151 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:31.003+0000 12a5a640 -1 osd.1 58 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:31.422 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:31.413+0000 12a5a640 -1 osd.3 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 13.6 13.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:31.792 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:31.621+0000 12a5a640 -1 osd.2 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:32.017 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:31.996+0000 ea52640 -1 osd.0 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:32.022 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:31.996+0000 12a5a640 -1 osd.6 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:32.489 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:32.479+0000 12a5a640 -1 osd.3 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 13.6 13.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:32.901 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:32.781+0000 12a5a640 -1 osd.1 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:33.152 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:33.146+0000 12a5a640 -1 osd.6 60 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:33.321 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:33.282+0000 12a5a640 -1 osd.2 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:33.647 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:33.613+0000 ea52640 -1 osd.0 60 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:33.836 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:33.810+0000 12a5a640 -1 osd.3 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 13.6 13:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:33.928 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:33.890+0000 12a5a640 -1 osd.1 60 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:33.999 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 1 -- 172.21.15.120:0/2 api_misc: [==========] Running 12 tests from 4 test suites. 2024-02-06T21:43:33.999 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [----------] Global test environment set-up. 2024-02-06T21:43:33.999 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-02-06T21:43:33.999 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-02-06T21:43:33.999 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-02-06T21:43:33.999 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-02-06T21:43:33.999 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: 2024-02-06T21:43:33.999 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-02-06T21:43:33.999 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-02-06T21:43:34.000 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-02-06T21:43:34.000 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: 2024-02-06T21:42:35.817+0000 7f1eb4042780 -1 failed for service _ceph-mon._tcp 2024-02-06T21:43:34.000 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: 2024-02-06T21:42:35.817+0000 7f1eb4042780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-02-06T21:43:34.000 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (225 ms) 2024-02-06T21:43:34.000 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-02-06T21:43:34.000 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2050 ms) 2024-02-06T21:43:34.000 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2275 ms total) 2024-02-06T21:43:34.000 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: 2024-02-06T21:43:34.000 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-02-06T21:43:34.000 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-02-06T21:43:34.000 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3fecc1720 2024-02-06T21:43:34.000 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3fef04990 2024-02-06T21:43:34.000 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3fee2d5e0 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3fecc57c0 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3fee42310 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3fee42490 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff035c70 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff036650 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff037050 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff037aa0 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff02af20 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3fef924b0 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff038e00 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff037800 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff039d10 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff038bb0 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff036800 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff033ad0 2024-02-06T21:43:34.001 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff036910 2024-02-06T21:43:34.002 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff036a20 2024-02-06T21:43:34.002 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff036b30 2024-02-06T21:43:34.002 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff0342d0 2024-02-06T21:43:34.002 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff033be0 2024-02-06T21:43:34.002 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff033cf0 2024-02-06T21:43:34.002 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff033e60 2024-02-06T21:43:34.002 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff035d80 2024-02-06T21:43:34.002 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff035e90 2024-02-06T21:43:34.002 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff035fa0 2024-02-06T21:43:34.002 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff0360b0 2024-02-06T21:43:34.002 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff039e40 2024-02-06T21:43:34.002 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff03a430 2024-02-06T21:43:34.002 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff03a540 2024-02-06T21:43:34.002 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff03a650 2024-02-06T21:43:34.003 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff037160 2024-02-06T21:43:34.003 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff0372b0 2024-02-06T21:43:34.003 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff037420 2024-02-06T21:43:34.003 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff037530 2024-02-06T21:43:34.003 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff039f50 2024-02-06T21:43:34.003 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff03a060 2024-02-06T21:43:34.003 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff03a170 2024-02-06T21:43:34.003 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff03a280 2024-02-06T21:43:34.003 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff034b80 2024-02-06T21:43:34.003 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff034c90 2024-02-06T21:43:34.003 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff034e00 2024-02-06T21:43:34.003 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff034f10 2024-02-06T21:43:34.003 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff038f30 2024-02-06T21:43:34.003 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff0353a0 2024-02-06T21:43:34.003 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff039980 2024-02-06T21:43:34.004 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff039ad0 2024-02-06T21:43:34.004 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff0354b0 2024-02-06T21:43:34.004 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff0355c0 2024-02-06T21:43:34.474 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:34.462+0000 12a5a640 -1 osd.2 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:34.752 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:34.674+0000 12a5a640 -1 osd.6 61 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:34.867 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:34.856+0000 ea52640 -1 osd.0 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:34.876 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:34.855+0000 12a5a640 -1 osd.3 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 13.6 13:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:35.208 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:35.201+0000 12a5a640 -1 osd.1 60 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:35.620 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:35.598+0000 12a5a640 -1 osd.2 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:35.869 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:03:15.683 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:35.905 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:35.763+0000 12a5a640 -1 osd.6 62 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:35.920 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:35.911+0000 12a5a640 -1 osd.3 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 13.6 13:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:35.948 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:03:15.763 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:35.956 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:03:15.771 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:36.020 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:03:15.783 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:36.039 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:03:15.854 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:36.039 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:03:15.803 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:36.175 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:03:15.990 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:36.195 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:03:15.959 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:36.195 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:03:16.010 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:36.250 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:36.182+0000 ea52640 -1 osd.0 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:36.572 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:36.380+0000 12a5a640 -1 osd.1 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:36.773 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:36.718+0000 12a5a640 -1 osd.2 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:36.883 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:36.826+0000 12a5a640 -1 osd.6 62 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:37.073 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:36.995+0000 12a5a640 -1 osd.3 63 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4435.0:2 13.6 13:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:37.782 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:37.677+0000 12a5a640 -1 osd.1 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:37.814 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:37.754+0000 12a5a640 -1 osd.2 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:38.108 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:38.090+0000 12a5a640 -1 osd.3 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4637.0:2 60.15 60:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-02-06T21:43:38.113 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:38.077+0000 ea52640 -1 osd.0 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:38.247 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:38.133+0000 12a5a640 -1 osd.6 63 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:38.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: starte api_pool: Running main() from gmock_main.cc 2024-02-06T21:43:38.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-02-06T21:43:38.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [----------] Global test environment set-up. 2024-02-06T21:43:38.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-02-06T21:43:38.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-02-06T21:43:38.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ OK ] LibRadosPools.PoolList (7707 ms) 2024-02-06T21:43:38.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-02-06T21:43:38.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3831 ms) 2024-02-06T21:43:38.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-02-06T21:43:38.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (6592 ms) 2024-02-06T21:43:38.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-02-06T21:43:38.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (4706 ms) 2024-02-06T21:43:38.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-02-06T21:43:38.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (4325 ms) 2024-02-06T21:43:38.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-02-06T21:43:38.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (10788 ms) 2024-02-06T21:43:38.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-02-06T21:43:38.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (7159 ms) 2024-02-06T21:43:38.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-02-06T21:43:38.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (7372 ms) 2024-02-06T21:43:38.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-02-06T21:43:38.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (10610 ms) 2024-02-06T21:43:38.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [----------] 9 tests from LibRadosPools (63091 ms total) 2024-02-06T21:43:38.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: 2024-02-06T21:43:38.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [----------] Global test environment tear-down 2024-02-06T21:43:38.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (63104 ms total) 2024-02-06T21:43:38.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_pool: [ PASSED ] 9 tests. 2024-02-06T21:43:38.905 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:38.805+0000 12a5a640 -1 osd.2 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:38.983 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:38.949+0000 12a5a640 -1 osd.1 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:39.174 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:39.159+0000 12a5a640 -1 osd.6 64 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:39.490 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:39.472+0000 ea52640 -1 osd.0 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:40.348 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:40.296+0000 12a5a640 -1 osd.2 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:40.589 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:40.488+0000 12a5a640 -1 osd.6 65 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:40.613 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:40.597+0000 ea52640 -1 osd.0 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:41.008 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:40.878+0000 12a5a640 -1 osd.1 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:41.428 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:41.419+0000 12a5a640 -1 osd.2 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:41.620 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:41.614+0000 ea52640 -1 osd.0 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:41.640 INFO:tasks.workunit.client.0.smithi120.stdout:499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:6 12.1 12:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57) v8 -- 0x55ef77a29cf0 con 0x55ef778a7460 2024-02-06T21:43:41.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 5 2024-02-06T21:43:41.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:30.881+0000 7fc1af40c780 5 client.4426.objecter 2 in flight 2024-02-06T21:43:41.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:31.637+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:41.640 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:31.637+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:41.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:31.637+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:41.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:32.638+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:41.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:32.638+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:41.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:32.638+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:41.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:33.638+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:41.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:33.638+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:41.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:33.638+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:41.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:34.638+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:41.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:34.638+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:41.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:34.638+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:41.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:35.638+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:41.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:35.638+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:41.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:35.638+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:41.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:35.694+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:43:41.641 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:36.638+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:36.638+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:36.638+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:37.639+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:37.639+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:37.639+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:38.639+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:38.639+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:38.639+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:38.639+0000 7fc1a8c33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:39.639+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:39.639+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:39.639+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:40.639+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:40.639+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:40.639+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:41.642 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:40.695+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:43:41.900 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:41.797+0000 12a5a640 -1 osd.6 66 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:42.268 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:42.161+0000 12a5a640 -1 osd.1 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:42.604 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:42.502+0000 12a5a640 -1 osd.2 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:42.931 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:42.855+0000 ea52640 -1 osd.0 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:43.212 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:43.199+0000 12a5a640 -1 osd.6 67 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:43.533 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:43.525+0000 12a5a640 -1 osd.2 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:44.390 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:44.295+0000 12a5a640 -1 osd.1 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:44.463 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:44.453+0000 ea52640 -1 osd.0 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:44.492 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:44.290+0000 12a5a640 -1 osd.6 69 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:45.304 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:45.299+0000 12a5a640 -1 osd.6 69 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:45.699 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:45.524+0000 12a5a640 -1 osd.1 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:45.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:41.639+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:45.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:41.639+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:45.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:41.639+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:45.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:42.639+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:45.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:42.640+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:45.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:42.640+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:45.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:43.640+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:45.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:43.640+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:45.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:43.640+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:45.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:44.640+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:45.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:44.640+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:45.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:44.640+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:45.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.640+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:45.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.640+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:45.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.640+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:45.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.696+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:43:45.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.696+0000 7fc1ae69e640 2 client.4426.objecter tid 5 on osd.1 is laggy 2024-02-06T21:43:45.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.696+0000 7fc1ae69e640 2 client.4426.objecter tid 6 on osd.1 is laggy 2024-02-06T21:43:45.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.696+0000 7fc1ae69e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:43:45.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.696+0000 7fc1ae69e640 10 monclient: _renew_subs 2024-02-06T21:43:45.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.696+0000 7fc1ae69e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:43:45.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.696+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=58}) v3 -- 0x7fc1840038c0 con 0x55ef778e6b30 2024-02-06T21:43:45.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.696+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- ping magic: 0 v1 -- 0x7fc1840042b0 con 0x55ef778a7460 2024-02-06T21:43:45.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 17 ==== osd_map(58..71 src has 1..71) v4 ==== 11884+0+0 (secure 0 0 0) 0x7fc1a005d530 con 0x55ef778e6b30 2024-02-06T21:43:45.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(58..71 src has 1..71) v4 2024-02-06T21:43:45.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [58,71] > 57 2024-02-06T21:43:45.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 58 2024-02-06T21:43:45.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:45.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 58 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:45.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:45.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 58 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:45.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 59 2024-02-06T21:43:45.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:45.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 59 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:45.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:45.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 59 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:45.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 60 2024-02-06T21:43:45.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 60 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 60 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 61 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 61 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 61 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 62 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 62 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:45.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:45.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 62 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:45.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 63 2024-02-06T21:43:45.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:45.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 63 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:45.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:45.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 63 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:45.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 64 2024-02-06T21:43:45.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:45.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 64 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.707 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:45.708 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:45.708 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 64 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.708 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:45.708 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 65 2024-02-06T21:43:45.708 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:45.708 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 65 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.708 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:45.711 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:45.711 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 65 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.712 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:45.712 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 66 2024-02-06T21:43:45.712 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:45.712 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 66 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.712 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:45.712 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:45.712 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 66 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.712 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:45.712 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 67 2024-02-06T21:43:45.712 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:45.712 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 67 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.712 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:45.712 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:45.712 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 67 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 68 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 68 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 68 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 69 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 69 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 69 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:45.713 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 70 2024-02-06T21:43:45.714 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:45.714 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 70 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.714 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:45.714 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:45.714 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 70 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.714 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:45.714 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.701+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 71 2024-02-06T21:43:45.714 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.702+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:45.714 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 71 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:45.714 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:45.830 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:45.732+0000 ea52640 -1 osd.0 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:46.149 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:46.115+0000 12a5a640 -1 osd.2 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:46.571 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T21:43:46.466+0000 12a5a640 -1 osd.3 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4637.0:2 60.15 60:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-02-06T21:43:46.736 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:46.626+0000 12a5a640 -1 osd.6 71 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:46.966 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:46.920+0000 ea52640 -1 osd.0 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:47.559 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:47.455+0000 12a5a640 -1 osd.1 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:47.628 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:43:47.607+0000 12a5a640 -1 osd.2 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4539.0:1 40.6s0 40:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-02-06T21:43:47.702 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:47.695+0000 12a5a640 -1 osd.6 71 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:48.095 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:47.989+0000 ea52640 -1 osd.0 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:48.755 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:48.549+0000 12a5a640 -1 osd.1 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:49.072 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T21:43:48.996+0000 12a5a640 -1 osd.6 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:2 17.1e 17:7d0ab5cf:test-rados-api-smithi120-40515-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:49.798 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:49.785+0000 12a5a640 -1 osd.1 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:49.866 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:49.795+0000 ea52640 -1 osd.0 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:50.700 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.702+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:50.700 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 71 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:50.700 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:50.700 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.702+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:43:50.700 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.702+0000 7fc1a9c35640 20 client.4426.objecter 5 12.d3a59201 osd.1 exist [read 0~256] 2024-02-06T21:43:50.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:45.702+0000 7fc1a9c35640 20 client.4426.objecter 6 12.a7bbaa1 osd.1 noexist [read 0~256] 2024-02-06T21:43:50.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:46.640+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:50.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:46.640+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:50.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:46.640+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:50.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:47.640+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:50.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:47.640+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:50.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:47.640+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:50.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:48.639+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:50.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:48.639+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:50.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:48.639+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:50.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:48.639+0000 7fc1a8c33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:43:50.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:49.639+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:50.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:49.639+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:50.701 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:49.639+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:50.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.639+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:50.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.639+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:50.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.639+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:50.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.696+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:43:50.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.696+0000 7fc1ae69e640 2 client.4426.objecter tid 5 on osd.1 is laggy 2024-02-06T21:43:50.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.696+0000 7fc1ae69e640 2 client.4426.objecter tid 6 on osd.1 is laggy 2024-02-06T21:43:50.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.696+0000 7fc1ae69e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:43:50.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.696+0000 7fc1ae69e640 10 monclient: _renew_subs 2024-02-06T21:43:50.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.696+0000 7fc1ae69e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:43:50.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.696+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=72}) v3 -- 0x7fc1840038c0 con 0x55ef778e6b30 2024-02-06T21:43:50.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.696+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- ping magic: 0 v1 -- 0x7fc184004270 con 0x55ef778a7460 2024-02-06T21:43:50.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 18 ==== osd_map(72..75 src has 1..75) v4 ==== 2439+0+0 (secure 0 0 0) 0x7fc1a005d530 con 0x55ef778e6b30 2024-02-06T21:43:50.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(72..75 src has 1..75) v4 2024-02-06T21:43:50.702 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [72,75] > 71 2024-02-06T21:43:50.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 72 2024-02-06T21:43:50.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:50.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 72 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:50.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:50.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:50.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 72 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:50.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:50.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 73 2024-02-06T21:43:50.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:50.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 73 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:50.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:50.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:50.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 73 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:50.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:50.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 74 2024-02-06T21:43:50.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:50.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 74 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:50.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:50.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:50.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 74 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:50.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:50.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 75 2024-02-06T21:43:50.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:50.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 75 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:50.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:50.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:50.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 75 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:50.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:51.007 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:50.948+0000 12a5a640 -1 osd.1 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:51.314 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:51.302+0000 ea52640 -1 osd.0 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:52.169 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:52.073+0000 12a5a640 -1 osd.1 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:52.905 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:52.887+0000 ea52640 -1 osd.0 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:53.871 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:53.759+0000 12a5a640 -1 osd.1 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:54.371 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:54.304+0000 ea52640 -1 osd.0 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:54.895 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:54.810+0000 12a5a640 -1 osd.1 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:55.604 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:55.508+0000 ea52640 -1 osd.0 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:55.715 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:43:55.715 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter 5 12.d3a59201 osd.1 exist [read 0~256] 2024-02-06T21:43:55.715 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:50.698+0000 7fc1a9c35640 20 client.4426.objecter 6 12.a7bbaa1 osd.1 noexist [read 0~256] 2024-02-06T21:43:55.715 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:51.639+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:55.715 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:51.639+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:55.715 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:51.639+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:55.716 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:52.639+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:55.716 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:52.639+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:55.716 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:52.639+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:55.716 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:53.639+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:55.716 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:53.639+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:55.716 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:53.639+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:55.716 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:54.639+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:55.716 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:54.639+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:55.716 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:54.639+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:55.716 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.640+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:43:55.716 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.640+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:43:55.717 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.640+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:43:55.717 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.697+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:43:55.717 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.697+0000 7fc1ae69e640 2 client.4426.objecter tid 5 on osd.1 is laggy 2024-02-06T21:43:55.717 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.697+0000 7fc1ae69e640 2 client.4426.objecter tid 6 on osd.1 is laggy 2024-02-06T21:43:55.717 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.697+0000 7fc1ae69e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:43:55.717 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.697+0000 7fc1ae69e640 10 monclient: _renew_subs 2024-02-06T21:43:55.717 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.697+0000 7fc1ae69e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:43:55.717 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.697+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=76}) v3 -- 0x7fc1840038c0 con 0x55ef778e6b30 2024-02-06T21:43:55.717 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.697+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- ping magic: 0 v1 -- 0x7fc1840042d0 con 0x55ef778a7460 2024-02-06T21:43:55.717 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 19 ==== osd_map(76..80 src has 1..80) v4 ==== 5270+0+0 (secure 0 0 0) 0x7fc1a005d530 con 0x55ef778e6b30 2024-02-06T21:43:55.717 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(76..80 src has 1..80) v4 2024-02-06T21:43:55.717 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [76,80] > 75 2024-02-06T21:43:55.717 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 76 2024-02-06T21:43:55.717 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:55.718 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 76 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:55.718 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:55.718 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:55.718 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 76 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:55.718 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:55.718 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 77 2024-02-06T21:43:55.718 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:55.718 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 77 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:55.718 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:55.718 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:55.718 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 77 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:55.718 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:55.718 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 78 2024-02-06T21:43:55.719 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:55.719 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 78 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:55.719 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:55.719 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:55.719 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 78 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:55.719 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:55.719 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 79 2024-02-06T21:43:55.719 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:55.719 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 79 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:55.719 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:55.719 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:55.719 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 79 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:55.719 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:43:55.719 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 80 2024-02-06T21:43:55.720 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:43:55.720 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 80 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:43:55.720 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:43:55.720 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:43:56.058 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:03:35.872 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:56.130 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:03:35.945 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:56.178 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:56.169+0000 12a5a640 -1 osd.1 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:56.184 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:03:35.948 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:56.186 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:03:35.950 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:56.200 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:03:36.015 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:56.206 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:03:36.021 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:56.288 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:03:36.103 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:56.299 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:03:36.063 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:56.335 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:03:36.150 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:43:56.746 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:56.559+0000 ea52640 -1 osd.0 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:57.458 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:57.360+0000 12a5a640 -1 osd.1 79 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:57.873 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 clied 0x55a3ff0356d0 2024-02-06T21:43:57.873 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff0357e0 2024-02-06T21:43:57.873 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff039040 2024-02-06T21:43:57.873 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff0394b0 2024-02-06T21:43:57.873 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff0343e0 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff0344f0 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff034600 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3fee28150 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff0395c0 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 0x55a3ff039730 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: started 61 aios 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3fecc1720 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3fef04990 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3fee2d5e0 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3fecc57c0 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3fee42310 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3fee42490 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff035c70 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff036650 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff037050 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff037aa0 2024-02-06T21:43:57.874 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff02af20 2024-02-06T21:43:57.875 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3fef924b0 2024-02-06T21:43:57.875 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff038e00 2024-02-06T21:43:57.875 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff037800 2024-02-06T21:43:57.875 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff039d10 2024-02-06T21:43:57.875 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff038bb0 2024-02-06T21:43:57.875 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff036800 2024-02-06T21:43:57.875 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff033ad0 2024-02-06T21:43:57.875 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff036910 2024-02-06T21:43:57.875 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff036a20 2024-02-06T21:43:57.875 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff036b30 2024-02-06T21:43:57.875 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff0342d0 2024-02-06T21:43:57.875 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff033be0 2024-02-06T21:43:57.875 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff033cf0 2024-02-06T21:43:57.875 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff033e60 2024-02-06T21:43:57.875 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff035d80 2024-02-06T21:43:57.876 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff035e90 2024-02-06T21:43:57.876 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff035fa0 2024-02-06T21:43:57.876 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff0360b0 2024-02-06T21:43:57.876 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff039e40 2024-02-06T21:43:57.876 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff03a430 2024-02-06T21:43:57.876 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff03a540 2024-02-06T21:43:57.876 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff03a650 2024-02-06T21:43:57.876 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff037160 2024-02-06T21:43:57.876 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff0372b0 2024-02-06T21:43:57.876 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff037420 2024-02-06T21:43:57.876 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff037530 2024-02-06T21:43:57.876 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff039f50 2024-02-06T21:43:57.876 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff03a060 2024-02-06T21:43:57.876 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff03a170 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff03a280 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff034b80 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff034c90 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff034e00 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff034f10 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff038f30 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff0353a0 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff039980 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff039ad0 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff0354b0 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff0355c0 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff0356d0 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff0357e0 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff039040 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff0394b0 2024-02-06T21:43:57.877 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff0343e0 2024-02-06T21:43:57.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff0344f0 2024-02-06T21:43:57.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff034600 2024-02-06T21:43:57.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3fee28150 2024-02-06T21:43:57.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff0395c0 2024-02-06T21:43:57.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: waiting 0x55a3ff039730 2024-02-06T21:43:57.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: done. 2024-02-06T21:43:57.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (70513 ms) 2024-02-06T21:43:57.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (70513 ms total) 2024-02-06T21:43:57.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: 2024-02-06T21:43:57.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-02-06T21:43:57.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-02-06T21:43:57.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-02-06T21:43:57.878 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-02-06T21:43:58.014 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:57.917+0000 ea52640 -1 osd.0 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:58.604 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:58.595+0000 12a5a640 -1 osd.1 81 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:43:59.095 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:43:59.074+0000 ea52640 -1 osd.0 79 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:43:59.859 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:43:59.688+0000 12a5a640 -1 osd.1 82 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:00.603 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ OK ] Li watch_notify: Running main() from gmock_main.cc 2024-02-06T21:44:00.603 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-02-06T21:44:00.603 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: [----------] Global test environment set-up. 2024-02-06T21:44:00.603 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-02-06T21:44:00.603 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-02-06T21:44:00.603 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: handle_notify cookie 94617346696064 notify_id 253403070464 notifier_gid 4543 2024-02-06T21:44:00.604 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (60562 ms) 2024-02-06T21:44:00.604 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-02-06T21:44:00.604 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: Trying... 2024-02-06T21:44:00.604 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: handle_notify cookie 94617359170592 notify_id 330712481792 notifier_gid 4682 2024-02-06T21:44:00.604 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: Waiting for 3.000000000s 2024-02-06T21:44:00.604 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: Timed out. 2024-02-06T21:44:00.604 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: Flushing... 2024-02-06T21:44:00.604 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: Flushed... 2024-02-06T21:44:00.604 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (24013 ms) 2024-02-06T21:44:00.604 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (84575 ms total) 2024-02-06T21:44:00.604 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: 2024-02-06T21:44:00.604 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: [----------] Global test environment tear-down 2024-02-06T21:44:00.604 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (84575 ms total) 2024-02-06T21:44:00.604 INFO:tasks.workunit.client.0.smithi120.stdout: watch_notify: [ PASSED ] 2 tests. 2024-02-06T21:44:00.763 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:00.695+0000 12a5a640 -1 osd.1 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:00.766 INFO:tasks.workunit.client.0.smithi120.stdout:nt.4426.objecter _calc_target epoch 80 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:00.766 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:00.766 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:44:00.766 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter 5 12.d3a59201 osd.1 exist [read 0~256] 2024-02-06T21:44:00.766 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:55.713+0000 7fc1a9c35640 20 client.4426.objecter 6 12.a7bbaa1 osd.1 noexist [read 0~256] 2024-02-06T21:44:00.766 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:56.640+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:00.766 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:56.640+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:00.766 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:56.640+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:00.766 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:57.640+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:00.766 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:57.640+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:00.767 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:57.640+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:00.767 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:58.640+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:00.767 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:58.640+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:00.767 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:58.640+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:00.767 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:58.640+0000 7fc1a8c33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:44:00.767 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:59.640+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:00.767 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:59.640+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:00.767 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:43:59.640+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:00.767 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.641+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:00.767 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.641+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:00.767 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.641+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:00.767 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.698+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:44:00.767 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.698+0000 7fc1ae69e640 2 client.4426.objecter tid 5 on osd.1 is laggy 2024-02-06T21:44:00.767 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.698+0000 7fc1ae69e640 2 client.4426.objecter tid 6 on osd.1 is laggy 2024-02-06T21:44:00.767 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.698+0000 7fc1ae69e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:44:00.768 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.698+0000 7fc1ae69e640 10 monclient: _renew_subs 2024-02-06T21:44:00.768 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.698+0000 7fc1ae69e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:44:00.768 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.698+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=81}) v3 -- 0x7fc184003870 con 0x55ef778e6b30 2024-02-06T21:44:00.768 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.698+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- ping magic: 0 v1 -- 0x7fc1840042d0 con 0x55ef778a7460 2024-02-06T21:44:00.768 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 20 ==== osd_map(81..84 src has 1..84) v4 ==== 1914+0+0 (secure 0 0 0) 0x7fc1a005d530 con 0x55ef778e6b30 2024-02-06T21:44:00.768 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(81..84 src has 1..84) v4 2024-02-06T21:44:00.768 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [81,84] > 80 2024-02-06T21:44:00.768 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 81 2024-02-06T21:44:00.768 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:44:00.768 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 81 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:00.768 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:00.768 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:44:00.768 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 81 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:00.769 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:00.769 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 82 2024-02-06T21:44:00.769 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:44:00.769 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 82 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:00.769 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:00.769 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:44:00.769 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 82 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:00.769 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:00.769 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 83 2024-02-06T21:44:00.769 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:44:00.769 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 83 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:00.769 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:00.769 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:44:00.769 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 83 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:00.770 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:00.770 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 84 2024-02-06T21:44:00.770 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:44:00.770 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 84 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:00.770 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:00.770 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:44:00.770 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 84 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:00.770 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:00.770 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:44:00.770 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter 5 12.d3a59201 osd.1 exist [read 0~256] 2024-02-06T21:44:00.856 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:44:00.723+0000 ea52640 -1 osd.0 81 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:44:01.740 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:01.716+0000 12a5a640 -1 osd.1 84 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:02.941 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:02.828+0000 12a5a640 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:03.789 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:44:03.780+0000 ea52640 -1 osd.0 83 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:44:03.961 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:03.954+0000 12a5a640 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:04.973 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:04.919+0000 12a5a640 -1 osd.1 86 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:04.974 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:44:04.874+0000 ea52640 -1 osd.0 85 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:44:05.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:00.764+0000 7fc1a9c35640 20 client.4426.objecter 6 12.a7bbaa1 osd.1 noexist [read 0~256] 2024-02-06T21:44:05.703 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:01.641+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:05.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:01.641+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:05.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:01.641+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:05.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:02.643+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:05.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:02.643+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:05.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:02.643+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:05.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:03.643+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:05.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:03.643+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:05.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:03.643+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:05.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:04.643+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:05.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:04.643+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:05.704 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:04.643+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:05.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.643+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:05.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.643+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:05.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.643+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:05.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.699+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:44:05.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.699+0000 7fc1ae69e640 2 client.4426.objecter tid 5 on osd.1 is laggy 2024-02-06T21:44:05.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.699+0000 7fc1ae69e640 2 client.4426.objecter tid 6 on osd.1 is laggy 2024-02-06T21:44:05.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.699+0000 7fc1ae69e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:44:05.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.699+0000 7fc1ae69e640 10 monclient: _renew_subs 2024-02-06T21:44:05.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.699+0000 7fc1ae69e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:44:05.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.699+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=85}) v3 -- 0x7fc184003870 con 0x55ef778e6b30 2024-02-06T21:44:05.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.699+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- ping magic: 0 v1 -- 0x7fc1840042d0 con 0x55ef778a7460 2024-02-06T21:44:05.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 21 ==== osd_map(85..88 src has 1..88) v4 ==== 1444+0+0 (secure 0 0 0) 0x7fc1a005d530 con 0x55ef778e6b30 2024-02-06T21:44:05.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(85..88 src has 1..88) v4 2024-02-06T21:44:05.705 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [85,88] > 84 2024-02-06T21:44:05.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 85 2024-02-06T21:44:05.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:44:05.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 85 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:05.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:05.706 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:44:05.943 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:44:05.937+0000 ea52640 -1 osd.0 85 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:44:06.033 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:06.027+0000 12a5a640 -1 osd.1 87 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:07.134 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T21:44:07.100+0000 ea52640 -1 osd.0 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4430.0:2 35.6 35:65d9be3e:test-rados-api-smithi120-40694-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) 2024-02-06T21:44:07.226 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:07.128+0000 12a5a640 -1 osd.1 88 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:07.645 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 85 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:07.645 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:07.645 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 86 2024-02-06T21:44:07.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:44:07.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 86 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:07.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:07.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:44:07.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 86 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:07.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:07.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 87 2024-02-06T21:44:07.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:44:07.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 87 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:07.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:07.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:44:07.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 87 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:07.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:07.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 88 2024-02-06T21:44:07.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:44:07.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 88 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:07.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:07.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:44:07.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 88 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:07.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:07.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:44:07.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter 5 12.d3a59201 osd.1 exist [read 0~256] 2024-02-06T21:44:07.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:05.702+0000 7fc1a9c35640 20 client.4426.objecter 6 12.a7bbaa1 osd.1 noexist [read 0~256] 2024-02-06T21:44:07.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:06.643+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:07.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:06.643+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:07.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:06.643+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:07.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:07.643+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:07.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:07.643+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:08.413 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:08.393+0000 12a5a640 -1 osd.1 89 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:09.917 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:09.792+0000 12a5a640 -1 osd.1 91 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:10.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:07.643+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:10.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:08.643+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:10.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:08.643+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:10.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:08.643+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:10.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:08.643+0000 7fc1a8c33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:44:10.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:09.643+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:10.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:09.643+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:10.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:09.643+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:10.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.643+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:10.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.643+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:10.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.643+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:10.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.700+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:44:10.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.700+0000 7fc1ae69e640 2 client.4426.objecter tid 5 on osd.1 is laggy 2024-02-06T21:44:10.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.700+0000 7fc1ae69e640 2 client.4426.objecter tid 6 on osd.1 is laggy 2024-02-06T21:44:10.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.700+0000 7fc1ae69e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T21:44:10.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.700+0000 7fc1ae69e640 10 monclient: _renew_subs 2024-02-06T21:44:10.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.700+0000 7fc1ae69e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T21:44:10.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.700+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=89}) v3 -- 0x7fc184003870 con 0x55ef778e6b30 2024-02-06T21:44:10.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.700+0000 7fc1ae69e640 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- ping magic: 0 v1 -- 0x7fc1840042d0 con 0x55ef778a7460 2024-02-06T21:44:10.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== mon.1 v2:172.21.15.135:3300/0 22 ==== osd_map(89..94 src has 1..94) v4 ==== 2307+0+0 (secure 0 0 0) 0x7fc1a005d530 con 0x55ef778e6b30 2024-02-06T21:44:10.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(89..94 src has 1..94) v4 2024-02-06T21:44:10.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [89,94] > 88 2024-02-06T21:44:10.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 89 2024-02-06T21:44:10.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:44:10.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 89 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:10.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:10.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:44:10.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 89 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:10.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:10.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 90 2024-02-06T21:44:10.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:44:10.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 90 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:10.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:10.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:44:10.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 90 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:10.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:10.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.724+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 91 2024-02-06T21:44:10.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:44:10.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 91 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:10.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:10.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:44:10.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 91 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:10.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:10.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 92 2024-02-06T21:44:10.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:44:10.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 92 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:10.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:10.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:44:10.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 92 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:10.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:10.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 93 2024-02-06T21:44:10.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:44:10.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 93 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:10.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:10.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:44:10.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 93 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:10.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:10.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 94 2024-02-06T21:44:10.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 10 client.4426.objecter checking op 5 2024-02-06T21:44:10.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 94 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:12.391 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:12.316+0000 12a5a640 -1 osd.1 93 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:13.369 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:13.348+0000 12a5a640 -1 osd.1 94 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:13.737 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02- cls: Running main() from gmock_main.cc 2024-02-06T21:44:13.737 INFO:tasks.workunit.client.0.smithi120.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-02-06T21:44:13.737 INFO:tasks.workunit.client.0.smithi120.stdout: cls: [----------] Global test environment set-up. 2024-02-06T21:44:13.737 INFO:tasks.workunit.client.0.smithi120.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-02-06T21:44:13.737 INFO:tasks.workunit.client.0.smithi120.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-02-06T21:44:13.738 INFO:tasks.workunit.client.0.smithi120.stdout: cls: [ OK ] NeoRadosCls.DNE (42309 ms) 2024-02-06T21:44:13.738 INFO:tasks.workunit.client.0.smithi120.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-02-06T21:44:13.738 INFO:tasks.workunit.client.0.smithi120.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (55530 ms) 2024-02-06T21:44:13.738 INFO:tasks.workunit.client.0.smithi120.stdout: cls: [----------] 2 tests from NeoRadosCls (97839 ms total) 2024-02-06T21:44:13.738 INFO:tasks.workunit.client.0.smithi120.stdout: cls: 2024-02-06T21:44:13.738 INFO:tasks.workunit.client.0.smithi120.stdout: cls: [----------] Global test environment tear-down 2024-02-06T21:44:13.738 INFO:tasks.workunit.client.0.smithi120.stdout: cls: [==========] 2 tests from 1 test suite ran. (97839 ms total) 2024-02-06T21:44:13.738 INFO:tasks.workunit.client.0.smithi120.stdout: cls: [ PASSED ] 2 tests. 2024-02-06T21:44:14.911 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:14.832+0000 12a5a640 -1 osd.1 96 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:15.205 INFO:tasks.workunit.client.0.smithi120.stdout:06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:15.205 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 10 client.4426.objecter checking op 6 2024-02-06T21:44:15.205 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 94 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:15.206 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:15.206 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:44:15.206 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter 5 12.d3a59201 osd.1 exist [read 0~256] 2024-02-06T21:44:15.206 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:10.725+0000 7fc1a9c35640 20 client.4426.objecter 6 12.a7bbaa1 osd.1 noexist [read 0~256] 2024-02-06T21:44:15.206 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:11.643+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:15.206 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:11.643+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:15.206 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:11.643+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:15.206 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:12.643+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:15.206 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:12.643+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:15.206 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:12.643+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:15.206 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:13.643+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:15.206 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:13.643+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:15.206 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:13.643+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:15.207 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:14.643+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:15.207 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:14.643+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:15.207 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:14.644+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:15.207 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 6 ==== osd_map(58..59 src has 1..60) v4 ==== 944+0+0 (crc 0 0 0) 0x7fc1a4062040 con 0x55ef778a7460 2024-02-06T21:44:15.207 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 7 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:15.207 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(58..59 src has 1..60) v4 2024-02-06T21:44:15.207 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-02-06T21:44:15.207 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:15.207 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map ignoring epochs [58,59] <= 94 2024-02-06T21:44:15.207 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:44:15.207 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1a9c35640 20 client.4426.objecter 5 12.d3a59201 osd.1 exist [read 0~256] 2024-02-06T21:44:15.207 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1a9c35640 20 client.4426.objecter 6 12.a7bbaa1 osd.1 noexist [read 0~256] 2024-02-06T21:44:15.207 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 5 ondisk uv 1 in 12.1 attempt 0 2024-02-06T21:44:15.208 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 5 2024-02-06T21:44:15.208 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 5 2024-02-06T21:44:15.208 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 5 2024-02-06T21:44:15.208 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 5 2024-02-06T21:44:15.208 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 5 2024-02-06T21:44:15.208 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 5 client.4426.objecter 1 in flight 2024-02-06T21:44:15.208 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 8 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:15.208 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-02-06T21:44:15.208 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:15.208 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 6 ondisk uv 0 in 12.1 attempt 0 2024-02-06T21:44:15.208 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 10 client.4426.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-02-06T21:44:15.208 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 0 2024-02-06T21:44:15.208 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 6 2024-02-06T21:44:15.209 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 6 2024-02-06T21:44:15.209 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:15.209 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 6 2024-02-06T21:44:15.209 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1acc3b640 5 client.4426.objecter 0 in flight 2024-02-06T21:44:15.209 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (44323 ms) 2024-02-06T21:44:15.209 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-02-06T21:44:15.209 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef778a6fc0 2024-02-06T21:44:15.209 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 94 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:15.209 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:15.209 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:44:15.209 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:15.209 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid exist '@12' '@12' [read 0~256] tid 7 osd.1 2024-02-06T21:44:15.209 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:15.210 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 7 2024-02-06T21:44:15.210 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 15 client.4426.objecter _send_op 7 to 12.1 on osd.1 2024-02-06T21:44:15.210 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:7 12.1 12:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v8 -- 0x55ef77987b70 con 0x55ef778a7460 2024-02-06T21:44:15.210 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:15.210 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 5 client.4426.objecter 1 in flight 2024-02-06T21:44:16.129 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:16.001+0000 12a5a640 -1 osd.1 97 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:18.326 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef77987f50 2024-02-06T21:44:18.326 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 94 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:18.327 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:18.327 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:44:18.327 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:18.327 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid noexist '@12' '@12' [read 0~256] tid 8 osd.1 2024-02-06T21:44:18.327 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:18.327 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 8 2024-02-06T21:44:18.327 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 15 client.4426.objecter _send_op 8 to 12.1 on osd.1 2024-02-06T21:44:18.327 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.203+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:8 12.1 12:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v8 -- 0x55ef779883f0 con 0x55ef778a7460 2024-02-06T21:44:18.327 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.204+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 5 2024-02-06T21:44:18.327 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.204+0000 7fc1af40c780 5 client.4426.objecter 2 in flight 2024-02-06T21:44:18.327 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.644+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:18.327 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.644+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:18.328 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.644+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:18.328 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:15.701+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:44:18.328 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:16.644+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:18.328 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:16.644+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:18.328 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:16.644+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:18.328 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:17.644+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:18.328 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:17.644+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:18.328 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:17.644+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:18.328 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 9 ==== osd_map(95..97 src has 1..97) v4 ==== 1198+0+0 (crc 0 0 0) 0x7fc1a40544d0 con 0x55ef778a7460 2024-02-06T21:44:18.328 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(95..97 src has 1..97) v4 2024-02-06T21:44:18.328 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 10 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:18.328 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-02-06T21:44:18.328 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [95,97] > 94 2024-02-06T21:44:18.328 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:18.329 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 95 2024-02-06T21:44:18.329 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 10 client.4426.objecter checking op 7 2024-02-06T21:44:18.329 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 95 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:18.329 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:18.329 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 10 client.4426.objecter checking op 8 2024-02-06T21:44:18.329 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 95 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:18.329 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:18.329 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 96 2024-02-06T21:44:18.329 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 10 client.4426.objecter checking op 7 2024-02-06T21:44:18.329 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 96 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:18.329 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:18.329 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 10 client.4426.objecter checking op 8 2024-02-06T21:44:18.329 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 96 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:18.330 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:18.330 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 97 2024-02-06T21:44:18.330 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 10 client.4426.objecter checking op 7 2024-02-06T21:44:18.330 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 97 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:18.330 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:18.330 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 10 client.4426.objecter checking op 8 2024-02-06T21:44:18.330 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 97 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:18.330 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:18.330 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:44:18.330 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 20 client.4426.objecter 7 12.d3a59201 osd.1 exist [read 0~256] 2024-02-06T21:44:18.330 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1a9c35640 20 client.4426.objecter 8 12.a7bbaa1 osd.1 noexist [read 0~256] 2024-02-06T21:44:18.330 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.324+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 7 ondisk uv 1 in 12.1 attempt 0 2024-02-06T21:44:18.330 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 5 2024-02-06T21:44:18.330 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 7 2024-02-06T21:44:18.330 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 7 2024-02-06T21:44:18.331 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 5 2024-02-06T21:44:18.331 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 7 2024-02-06T21:44:18.331 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 5 client.4426.objecter 1 in flight 2024-02-06T21:44:18.645 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 11 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:18.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-02-06T21:44:18.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:18.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 8 ondisk uv 0 in 12.1 attempt 0 2024-02-06T21:44:18.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 10 client.4426.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-02-06T21:44:18.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 0 2024-02-06T21:44:18.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 8 2024-02-06T21:44:18.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 8 2024-02-06T21:44:18.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:18.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 8 2024-02-06T21:44:18.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1acc3b640 5 client.4426.objecter 0 in flight 2024-02-06T21:44:18.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (3121 ms) 2024-02-06T21:44:18.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-02-06T21:44:18.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1af40c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-02-06T21:44:18.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1af40c780 20 librados: queue_aio_write 0x55ef777f9250 completion 0x55ef777f77c0 write_seq 1 2024-02-06T21:44:18.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef778a6fc0 2024-02-06T21:44:18.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 97 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:18.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1af40c780 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:18.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:44:18.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:18.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid exist '@12' '@12' [write 0~5 in=5b] tid 9 osd.1 2024-02-06T21:44:18.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:18.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 9 2024-02-06T21:44:18.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1af40c780 15 client.4426.objecter _send_op 9 to 12.1 on osd.1 2024-02-06T21:44:18.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:9 12.1 12:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v8 -- 0x55ef77987ed0 con 0x55ef778a7460 2024-02-06T21:44:18.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:18.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1af40c780 5 client.4426.objecter 1 in flight 2024-02-06T21:44:18.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.325+0000 7fc1af40c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-02-06T21:44:18.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.644+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:19.732 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:19.712+0000 12a5a640 -1 osd.1 100 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:20.902 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:20.889+0000 12a5a640 -1 osd.1 101 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:21.693 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:04:01.508 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:44:21.698 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:04:01.512 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:44:21.717 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:04:01.531 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:44:21.730 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:04:01.494 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:44:21.730 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:04:01.543 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:44:21.732 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:04:01.496 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:44:21.926 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:21.864+0000 12a5a640 -1 osd.1 102 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:22.031 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:04:01.795 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:44:22.050 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:04:01.865 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:44:22.055 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:04:01.870 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:44:23.267 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:23.162+0000 12a5a640 -1 osd.1 103 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:24.337 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:24.226+0000 12a5a640 -1 osd.1 104 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:24.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.644+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:24.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.644+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:24.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:18.644+0000 7fc1a8c33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:44:24.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.249+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 12 ==== osd_map(98..99 src has 1..100) v4 ==== 604+0+0 (crc 0 0 0) 0x7fc1a40544d0 con 0x55ef778a7460 2024-02-06T21:44:24.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.249+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(98..99 src has 1..100) v4 2024-02-06T21:44:24.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.249+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [98,99] > 97 2024-02-06T21:44:24.646 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.249+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 98 2024-02-06T21:44:24.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.249+0000 7fc1a9c35640 10 client.4426.objecter checking op 9 2024-02-06T21:44:24.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.249+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 98 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:24.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.249+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:24.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.249+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 99 2024-02-06T21:44:24.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.249+0000 7fc1a9c35640 10 client.4426.objecter checking op 9 2024-02-06T21:44:24.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.249+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 99 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:24.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.249+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:24.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.249+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:44:24.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.249+0000 7fc1a9c35640 20 client.4426.objecter 9 12.d3a59201 osd.1 exist [write 0~5 in=5b] 2024-02-06T21:44:24.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.644+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:24.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.644+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:24.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:19.644+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:24.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:20.644+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:24.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:20.644+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:24.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:20.644+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:24.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:20.702+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:44:24.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:21.644+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:24.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:21.644+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:24.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:21.644+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:24.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:22.645+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:24.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:22.645+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:24.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:22.645+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:24.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:23.645+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:24.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:23.645+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:24.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:23.645+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:24.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:24.645+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:24.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:24.645+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:25.682 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:25.596+0000 12a5a640 -1 osd.1 105 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:26.465 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02 api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-02-06T21:44:26.465 INFO:tasks.workunit.client.0.smithi120.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-02-06T21:44:26.465 INFO:tasks.workunit.client.0.smithi120.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-02-06T21:44:26.465 INFO:tasks.workunit.client.0.smithi120.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-02-06T21:44:26.465 INFO:tasks.workunit.client.0.smithi120.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (110623 ms) 2024-02-06T21:44:26.465 INFO:tasks.workunit.client.0.smithi120.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (110623 ms total) 2024-02-06T21:44:26.465 INFO:tasks.workunit.client.0.smithi120.stdout: api_cls_remote_reads: 2024-02-06T21:44:26.465 INFO:tasks.workunit.client.0.smithi120.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-02-06T21:44:26.465 INFO:tasks.workunit.client.0.smithi120.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (110623 ms total) 2024-02-06T21:44:26.465 INFO:tasks.workunit.client.0.smithi120.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-02-06T21:44:26.756 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:26.748+0000 12a5a640 -1 osd.1 106 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:26.827 INFO:tasks.workunit.client.0.smithi120.stdout:-06T21:44:24.645+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:25.645+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:25.645+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:25.645+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:25.703+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:44:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.646+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.646+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.646+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 13 ==== osd_op_reply(9 exist [write 0~5] v99'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(9 exist [write 0~5] v99'2 uv2 ondisk = 0) v8 2024-02-06T21:44:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 9 ondisk uv 2 in 12.1 attempt 0 2024-02-06T21:44:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 0 2024-02-06T21:44:26.828 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 9 2024-02-06T21:44:26.828 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 9 2024-02-06T21:44:26.828 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:26.828 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 9 2024-02-06T21:44:26.828 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1acc3b640 5 client.4426.objecter 0 in flight 2024-02-06T21:44:26.828 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1acc3b640 20 librados: complete_aio_write 0x55ef777f77c0 2024-02-06T21:44:26.828 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (8500 ms) 2024-02-06T21:44:26.828 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-02-06T21:44:26.828 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1af40c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-02-06T21:44:26.828 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1af40c780 20 librados: queue_aio_write 0x55ef777f9250 completion 0x55ef777e6ae0 write_seq 2 2024-02-06T21:44:26.828 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef778a6fc0 2024-02-06T21:44:26.828 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 99 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:26.828 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1af40c780 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:26.828 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:44:26.829 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:26.829 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid exist '@12' '@12' [write 0~5 in=5b] tid 10 osd.1 2024-02-06T21:44:26.829 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:26.829 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 10 2024-02-06T21:44:26.829 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1af40c780 15 client.4426.objecter _send_op 10 to 12.1 on osd.1 2024-02-06T21:44:27.021 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:10 12.1 12:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v8 -- 0x55ef77987ed0 con 0x55ef778a7460 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1af40c780 5 client.4426.objecter 1 in flight 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:26.826+0000 7fc1af40c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.020+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 14 ==== osd_map(100..107 src has 1..108) v4 ==== 11153+0+0 (crc 0 0 0) 0x7fc1a40544d0 con 0x55ef778a7460 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.020+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(100..107 src has 1..108) v4 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.020+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [100,107] > 99 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.020+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 100 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.020+0000 7fc1a9c35640 10 client.4426.objecter checking op 10 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.020+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 100 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.020+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.020+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 101 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.020+0000 7fc1a9c35640 10 client.4426.objecter checking op 10 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.020+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 101 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.020+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.020+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 102 2024-02-06T21:44:27.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 10 client.4426.objecter checking op 10 2024-02-06T21:44:27.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 102 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:27.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:27.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 103 2024-02-06T21:44:27.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 10 client.4426.objecter checking op 10 2024-02-06T21:44:27.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 103 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:27.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:27.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 104 2024-02-06T21:44:27.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 10 client.4426.objecter checking op 10 2024-02-06T21:44:27.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 104 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:27.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:28.118 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:28.038+0000 12a5a640 -1 osd.1 108 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:29.089 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:29.062+0000 12a5a640 -1 osd.1 109 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:30.589 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:30.502+0000 12a5a640 -1 osd.1 111 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:31.664 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:31.641+0000 12a5a640 -1 osd.1 111 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:32.743 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:32.733+0000 12a5a640 -1 osd.1 112 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:33.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 105 2024-02-06T21:44:33.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 10 client.4426.objecter checking op 10 2024-02-06T21:44:33.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 105 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:33.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:33.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 106 2024-02-06T21:44:33.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 10 client.4426.objecter checking op 10 2024-02-06T21:44:33.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 106 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:33.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:33.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 107 2024-02-06T21:44:33.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 10 client.4426.objecter checking op 10 2024-02-06T21:44:33.647 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 107 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:33.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:33.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:44:33.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.021+0000 7fc1a9c35640 20 client.4426.objecter 10 12.d3a59201 osd.1 exist [write 0~5 in=5b] 2024-02-06T21:44:33.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.646+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:33.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.646+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:33.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:27.646+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:33.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:28.646+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:33.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:28.646+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:33.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:28.646+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:33.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:28.646+0000 7fc1a8c33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:44:33.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:29.646+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:33.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:29.646+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:33.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:29.646+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:33.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:30.646+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:33.648 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:30.646+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:33.649 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:30.646+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:33.649 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:30.704+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:44:33.649 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:31.646+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:33.649 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:31.646+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:33.649 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:31.646+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:33.649 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:32.646+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:33.649 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:32.646+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:33.649 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:32.646+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:33.649 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:33.645+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:34.159 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:34.141+0000 12a5a640 -1 osd.1 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:35.188 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:35.181+0000 12a5a640 -1 osd.1 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:36.289 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:36.280+0000 12a5a640 -1 osd.1 116 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:37.368 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:37.304+0000 12a5a640 -1 osd.1 117 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:37.603 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:33.645+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:37.603 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:33.645+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:37.603 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:34.645+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:37.603 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:34.645+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:37.603 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:34.645+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:37.603 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:35.645+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:37.603 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:35.645+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:37.604 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:35.645+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:37.604 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:35.704+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:44:37.604 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:36.645+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:37.604 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:36.645+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:37.604 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:36.645+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:37.604 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 15 ==== osd_op_reply(10 exist [write 0~5] v107'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:37.604 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(10 exist [write 0~5] v107'3 uv3 ondisk = 0) v8 2024-02-06T21:44:37.604 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:37.604 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 10 ondisk uv 3 in 12.1 attempt 0 2024-02-06T21:44:37.604 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 0 2024-02-06T21:44:37.604 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 10 2024-02-06T21:44:37.604 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 10 2024-02-06T21:44:37.605 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:37.605 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 10 2024-02-06T21:44:37.605 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1acc3b640 5 client.4426.objecter 0 in flight 2024-02-06T21:44:37.605 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1acc3b640 20 librados: complete_aio_write 0x55ef777e6ae0 2024-02-06T21:44:37.605 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (10776 ms) 2024-02-06T21:44:37.605 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-02-06T21:44:37.605 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1af40c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-02-06T21:44:37.605 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1af40c780 20 librados: queue_aio_write 0x55ef777f9250 completion 0x55ef778a6fc0 write_seq 3 2024-02-06T21:44:37.605 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef7798b4f0 2024-02-06T21:44:37.605 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 107 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:37.605 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1af40c780 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:37.605 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:44:37.605 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:37.605 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid exist '@12' '@12' [write 0~5 in=5b] tid 11 osd.1 2024-02-06T21:44:38.201 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:38.201 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 11 2024-02-06T21:44:38.201 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1af40c780 15 client.4426.objecter _send_op 11 to 12.1 on osd.1 2024-02-06T21:44:38.201 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:11 12.1 12:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) v8 -- 0x55ef778a70d0 con 0x55ef778a7460 2024-02-06T21:44:38.201 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:38.201 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1af40c780 5 client.4426.objecter 1 in flight 2024-02-06T21:44:38.201 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.601+0000 7fc1af40c780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-02-06T21:44:38.201 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.645+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:38.202 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.645+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:38.202 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:37.645+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:38.202 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.198+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 16 ==== osd_map(108..118 src has 1..118) v4 ==== 9575+0+0 (crc 0 0 0) 0x7fc1a4054120 con 0x55ef778a7460 2024-02-06T21:44:38.202 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(108..118 src has 1..118) v4 2024-02-06T21:44:38.202 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [108,118] > 107 2024-02-06T21:44:38.202 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 108 2024-02-06T21:44:38.202 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 10 client.4426.objecter checking op 11 2024-02-06T21:44:38.202 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 108 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:38.202 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:38.202 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 109 2024-02-06T21:44:38.202 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 10 client.4426.objecter checking op 11 2024-02-06T21:44:38.202 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 109 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:38.203 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:38.203 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 110 2024-02-06T21:44:38.203 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 10 client.4426.objecter checking op 11 2024-02-06T21:44:38.203 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 110 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:38.203 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:38.203 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 111 2024-02-06T21:44:38.203 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 10 client.4426.objecter checking op 11 2024-02-06T21:44:38.203 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 111 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:38.203 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:38.203 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 112 2024-02-06T21:44:38.203 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 10 client.4426.objecter checking op 11 2024-02-06T21:44:38.203 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 112 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:38.203 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:38.204 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 113 2024-02-06T21:44:38.204 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 10 client.4426.objecter checking op 11 2024-02-06T21:44:38.204 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 113 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:38.204 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:38.204 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 114 2024-02-06T21:44:38.204 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 10 client.4426.objecter checking op 11 2024-02-06T21:44:38.204 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 114 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:38.204 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:38.204 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 115 2024-02-06T21:44:38.204 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 10 client.4426.objecter checking op 11 2024-02-06T21:44:38.204 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 115 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:38.204 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:38.204 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 116 2024-02-06T21:44:38.204 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 10 client.4426.objecter checking op 11 2024-02-06T21:44:38.205 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 116 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:38.205 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:38.205 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 117 2024-02-06T21:44:38.205 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 10 client.4426.objecter checking op 11 2024-02-06T21:44:38.205 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 117 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:38.205 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:38.205 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 118 2024-02-06T21:44:38.205 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 10 client.4426.objecter checking op 11 2024-02-06T21:44:38.205 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 118 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:38.205 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:38.205 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:44:38.834 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.199+0000 7fc1a9c35640 20 client.4426.objecter 11 12.d3a59201 osd.1 exist [write 0~5 in=5b] 2024-02-06T21:44:38.834 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.645+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:38.834 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.645+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:38.834 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.645+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:38.834 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.645+0000 7fc1a8c33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T21:44:38.834 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 17 ==== osd_op_reply(11 exist [write 0~5] v118'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:38.835 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(11 exist [write 0~5] v118'4 uv4 ondisk = 0) v8 2024-02-06T21:44:38.835 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:38.835 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 11 ondisk uv 4 in 12.1 attempt 0 2024-02-06T21:44:38.835 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 0 2024-02-06T21:44:38.835 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 11 2024-02-06T21:44:38.835 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 11 2024-02-06T21:44:38.835 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:38.835 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 11 2024-02-06T21:44:38.835 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1acc3b640 5 client.4426.objecter 0 in flight 2024-02-06T21:44:38.835 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1acc3b640 20 librados: complete_aio_write 0x55ef778a6fc0 2024-02-06T21:44:38.835 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (1231 ms) 2024-02-06T21:44:38.835 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-02-06T21:44:38.835 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef7798ae80 2024-02-06T21:44:38.836 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 118 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:38.836 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:38.836 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:44:38.836 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:38.836 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid exist '@12' '@12' [read 0~0] tid 12 osd.1 2024-02-06T21:44:38.836 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:38.836 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 12 2024-02-06T21:44:38.836 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 15 client.4426.objecter _send_op 12 to 12.1 on osd.1 2024-02-06T21:44:38.836 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:12 12.1 12:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e118) v8 -- 0x55ef778a70d0 con 0x55ef778a7460 2024-02-06T21:44:38.836 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:38.836 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 5 client.4426.objecter 1 in flight 2024-02-06T21:44:39.545 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef7798b320 2024-02-06T21:44:39.545 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 118 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:39.545 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:39.545 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:44:39.545 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:39.545 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid noexist '@12' '@12' [read 0~0] tid 13 osd.1 2024-02-06T21:44:39.545 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:39.545 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 13 2024-02-06T21:44:39.545 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 15 client.4426.objecter _send_op 13 to 12.1 on osd.1 2024-02-06T21:44:39.545 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:13 12.1 12:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e118) v8 -- 0x55ef77988750 con 0x55ef778a7460 2024-02-06T21:44:39.545 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 5 2024-02-06T21:44:39.545 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:38.832+0000 7fc1af40c780 5 client.4426.objecter 2 in flight 2024-02-06T21:44:39.546 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 19 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:39.546 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-02-06T21:44:39.546 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:39.546 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 12 ondisk uv 4 in 12.1 attempt 0 2024-02-06T21:44:39.546 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 5 2024-02-06T21:44:39.546 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 12 2024-02-06T21:44:39.546 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 12 2024-02-06T21:44:39.546 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 5 2024-02-06T21:44:39.546 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 12 2024-02-06T21:44:39.546 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 5 client.4426.objecter 1 in flight 2024-02-06T21:44:39.546 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 18 ==== osd_map(119..119 src has 1..119) v4 ==== 324+0+0 (crc 0 0 0) 0x7fc1a4054800 con 0x55ef778a7460 2024-02-06T21:44:39.546 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(119..119 src has 1..119) v4 2024-02-06T21:44:39.546 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [119,119] > 118 2024-02-06T21:44:39.546 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 119 2024-02-06T21:44:39.547 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 20 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:39.547 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-02-06T21:44:39.547 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:39.547 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1a9c35640 10 client.4426.objecter checking op 13 2024-02-06T21:44:39.547 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 119 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:39.547 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:39.547 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:44:39.547 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1a9c35640 20 client.4426.objecter 13 12.a7bbaa1 osd.1 noexist [read 0~0] 2024-02-06T21:44:39.547 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 13 ondisk uv 0 in 12.1 attempt 0 2024-02-06T21:44:39.547 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 10 client.4426.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-02-06T21:44:39.547 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 0 2024-02-06T21:44:39.547 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 13 2024-02-06T21:44:39.548 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 13 2024-02-06T21:44:39.548 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:39.548 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 13 2024-02-06T21:44:39.548 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1acc3b640 5 client.4426.objecter 0 in flight 2024-02-06T21:44:39.548 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (711 ms) 2024-02-06T21:44:39.548 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-02-06T21:44:39.548 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef7798ae80 2024-02-06T21:44:39.548 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 119 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:39.548 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:39.548 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:44:39.548 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:39.548 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid exist '@12' '@12' [read 0~0] tid 14 osd.1 2024-02-06T21:44:39.548 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:39.548 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 14 2024-02-06T21:44:39.549 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 15 client.4426.objecter _send_op 14 to 12.1 on osd.1 2024-02-06T21:44:39.549 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:14 12.1 12:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e119) v8 -- 0x55ef7798b320 con 0x55ef778a7460 2024-02-06T21:44:39.549 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:39.776 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:39.766+0000 12a5a640 -1 osd.1 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:39.925 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 5 client.4426.objecter 1 in flight 2024-02-06T21:44:39.925 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef77988750 2024-02-06T21:44:39.925 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 119 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:39.925 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:39.926 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:44:39.926 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:39.926 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid noexist '@12' '@12' [read 0~0] tid 15 osd.1 2024-02-06T21:44:39.926 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:39.926 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 15 2024-02-06T21:44:39.926 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 15 client.4426.objecter _send_op 15 to 12.1 on osd.1 2024-02-06T21:44:39.926 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:15 12.1 12:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e119) v8 -- 0x55ef77988bf0 con 0x55ef778a7460 2024-02-06T21:44:39.926 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 5 2024-02-06T21:44:39.926 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.543+0000 7fc1af40c780 5 client.4426.objecter 2 in flight 2024-02-06T21:44:39.926 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.646+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:39.926 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.646+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:39.926 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.646+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:39.926 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 21 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:39.926 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-02-06T21:44:39.927 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:39.927 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 14 ondisk uv 4 in 12.1 attempt 0 2024-02-06T21:44:39.927 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 5 2024-02-06T21:44:39.927 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 14 2024-02-06T21:44:39.927 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 14 2024-02-06T21:44:39.927 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 5 2024-02-06T21:44:39.927 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 14 2024-02-06T21:44:39.927 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 5 client.4426.objecter 1 in flight 2024-02-06T21:44:39.927 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 22 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:39.927 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-02-06T21:44:39.927 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:39.927 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 15 ondisk uv 0 in 12.1 attempt 0 2024-02-06T21:44:39.927 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 10 client.4426.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-02-06T21:44:39.927 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 0 2024-02-06T21:44:39.927 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 15 2024-02-06T21:44:39.928 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 15 2024-02-06T21:44:39.928 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:39.928 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 15 2024-02-06T21:44:39.928 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1acc3b640 5 client.4426.objecter 0 in flight 2024-02-06T21:44:39.928 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (381 ms) 2024-02-06T21:44:39.928 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-02-06T21:44:39.928 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef77988750 2024-02-06T21:44:39.928 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 119 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:39.928 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:39.928 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:44:39.928 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:39.928 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid exist '@12' '@12' [read 0~0] tid 16 osd.1 2024-02-06T21:44:39.928 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:39.929 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 16 2024-02-06T21:44:39.929 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 15 client.4426.objecter _send_op 16 to 12.1 on osd.1 2024-02-06T21:44:39.929 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:16 12.1 12:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e119) v8 -- 0x55ef77988bf0 con 0x55ef778a7460 2024-02-06T21:44:39.929 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:39.929 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 5 client.4426.objecter 1 in flight 2024-02-06T21:44:39.929 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef77989250 2024-02-06T21:44:39.929 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 119 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-02-06T21:44:39.929 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 20 client.4426.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-02-06T21:44:39.929 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:44:39.929 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:39.929 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid noexist '@12' '@12' [read 0~0] tid 17 osd.1 2024-02-06T21:44:40.351 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:40.352 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 17 2024-02-06T21:44:40.352 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 15 client.4426.objecter _send_op 17 to 12.1 on osd.1 2024-02-06T21:44:40.352 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:17 12.1 12:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e119) v8 -- 0x55ef7798bc40 con 0x55ef778a7460 2024-02-06T21:44:40.352 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 5 2024-02-06T21:44:40.352 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:39.923+0000 7fc1af40c780 5 client.4426.objecter 2 in flight 2024-02-06T21:44:40.352 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 23 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:40.352 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-02-06T21:44:40.352 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:40.352 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 16 ondisk uv 4 in 12.1 attempt 0 2024-02-06T21:44:40.352 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 5 2024-02-06T21:44:40.352 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 16 2024-02-06T21:44:40.352 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 16 2024-02-06T21:44:40.353 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 5 2024-02-06T21:44:40.353 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 16 2024-02-06T21:44:40.353 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 5 client.4426.objecter 1 in flight 2024-02-06T21:44:40.353 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 25 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:40.353 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-02-06T21:44:40.353 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:40.353 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 17 ondisk uv 0 in 12.1 attempt 0 2024-02-06T21:44:40.353 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 10 client.4426.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-02-06T21:44:40.353 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 24 ==== osd_map(120..120 src has 1..120) v4 ==== 296+0+0 (crc 0 0 0) 0x7fc1a4054800 con 0x55ef778a7460 2024-02-06T21:44:40.353 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 0 2024-02-06T21:44:40.353 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 17 2024-02-06T21:44:40.353 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 17 2024-02-06T21:44:40.353 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:40.353 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 17 2024-02-06T21:44:40.856 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:40.837+0000 12a5a640 -1 osd.1 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4584.0:2 44.6 44:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-02-06T21:44:43.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(120..120 src has 1..120) v4 2024-02-06T21:44:43.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1acc3b640 5 client.4426.objecter 0 in flight 2024-02-06T21:44:43.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [120,120] > 119 2024-02-06T21:44:43.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 120 2024-02-06T21:44:43.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (426 ms) 2024-02-06T21:44:43.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-02-06T21:44:43.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:44:43.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1af40c780 20 librados: queue_aio_write 0x55ef777f9250 completion 0x55ef7798bb30 write_seq 4 2024-02-06T21:44:43.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef77989250 2024-02-06T21:44:43.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 120 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:43.224 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1af40c780 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:43.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:44:43.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:43.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 18 osd.1 2024-02-06T21:44:43.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:43.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 18 2024-02-06T21:44:43.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1af40c780 15 client.4426.objecter _send_op 18 to 12.1 on osd.1 2024-02-06T21:44:43.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:18 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) v8 -- 0x55ef7798ae80 con 0x55ef778a7460 2024-02-06T21:44:43.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:43.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.350+0000 7fc1af40c780 5 client.4426.objecter 1 in flight 2024-02-06T21:44:43.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.646+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:43.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.646+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:43.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.646+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:43.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:40.705+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:44:43.225 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:41.646+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:43.226 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:41.646+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:43.226 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:41.646+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:43.226 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:42.646+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:43.226 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:42.646+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:43.226 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:42.646+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:43.348 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 26 ==== osd_op_reply(18 exist [writefull 0~5] v120'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:43.349 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(18 exist [writefull 0~5] v120'5 uv5 ondisk = 0) v8 2024-02-06T21:44:43.349 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:43.349 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 18 ondisk uv 5 in 12.1 attempt 0 2024-02-06T21:44:43.349 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 0 2024-02-06T21:44:43.349 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 18 2024-02-06T21:44:43.349 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 18 2024-02-06T21:44:43.349 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:43.349 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 18 2024-02-06T21:44:43.349 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1acc3b640 5 client.4426.objecter 0 in flight 2024-02-06T21:44:43.349 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1acc3b640 20 librados: complete_aio_write 0x55ef7798bb30 2024-02-06T21:44:43.349 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (2872 ms) 2024-02-06T21:44:43.349 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-02-06T21:44:43.349 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1af40c780 20 librados: queue_aio_write 0x55ef777f9250 completion 0x55ef778a71a0 write_seq 5 2024-02-06T21:44:43.350 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef77989230 2024-02-06T21:44:43.350 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 120 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:43.350 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1af40c780 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:43.350 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:44:43.350 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:43.350 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 19 osd.1 2024-02-06T21:44:43.350 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:43.350 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 19 2024-02-06T21:44:43.350 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1af40c780 15 client.4426.objecter _send_op 19 to 12.1 on osd.1 2024-02-06T21:44:43.350 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:19 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) v8 -- 0x55ef7798ae80 con 0x55ef778a7460 2024-02-06T21:44:43.350 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:43.350 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.222+0000 7fc1af40c780 5 client.4426.objecter 1 in flight 2024-02-06T21:44:43.350 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 27 ==== osd_map(121..123 src has 1..124) v4 ==== 2274+0+0 (crc 0 0 0) 0x7fc1a4054800 con 0x55ef778a7460 2024-02-06T21:44:43.351 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(121..123 src has 1..124) v4 2024-02-06T21:44:43.351 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [121,123] > 120 2024-02-06T21:44:44.195 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 121 2024-02-06T21:44:44.195 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 10 client.4426.objecter checking op 19 2024-02-06T21:44:44.195 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 121 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:44.195 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:44.195 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 122 2024-02-06T21:44:44.195 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 10 client.4426.objecter checking op 19 2024-02-06T21:44:44.195 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 122 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:44.195 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:44.196 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 123 2024-02-06T21:44:44.196 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 10 client.4426.objecter checking op 19 2024-02-06T21:44:44.196 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 123 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:44.196 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:44.196 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:44:44.196 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.347+0000 7fc1a9c35640 20 client.4426.objecter 19 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-02-06T21:44:44.196 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.646+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:44.196 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.646+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:44.196 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:43.646+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:44.196 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 28 ==== osd_op_reply(19 exist [writefull 0~5] v123'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:44.196 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(19 exist [writefull 0~5] v123'6 uv6 ondisk = 0) v8 2024-02-06T21:44:44.196 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:44.196 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 19 ondisk uv 6 in 12.1 attempt 0 2024-02-06T21:44:44.196 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 0 2024-02-06T21:44:44.197 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 19 2024-02-06T21:44:44.197 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 19 2024-02-06T21:44:44.197 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:44.197 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 19 2024-02-06T21:44:44.197 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1acc3b640 5 client.4426.objecter 0 in flight 2024-02-06T21:44:44.197 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1acc3b640 20 librados: complete_aio_write 0x55ef778a71a0 2024-02-06T21:44:44.197 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (971 ms) 2024-02-06T21:44:44.197 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-02-06T21:44:44.197 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1af40c780 20 librados: queue_aio_write 0x55ef777f9250 completion 0x55ef778a71a0 write_seq 6 2024-02-06T21:44:45.723 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1af40c780 10 client.4426.objecter _op_submit op 0x55ef77989230 2024-02-06T21:44:45.723 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1af40c780 20 client.4426.objecter _calc_target epoch 123 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:45.723 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1af40c780 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:45.723 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1af40c780 10 client.4426.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-02-06T21:44:45.723 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1af40c780 20 client.4426.objecter _get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:45.723 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1af40c780 10 client.4426.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 20 osd.1 2024-02-06T21:44:45.723 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1af40c780 20 client.4426.objecter get_session s=0x55ef77973370 osd=1 3 2024-02-06T21:44:45.723 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1af40c780 15 client.4426.objecter _session_op_assign 1 20 2024-02-06T21:44:45.724 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1af40c780 15 client.4426.objecter _send_op 20 to 12.1 on osd.1 2024-02-06T21:44:45.724 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- osd_op(unknown.0.0:20 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) v8 -- 0x55ef7798ae80 con 0x55ef778a7460 2024-02-06T21:44:45.724 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:45.724 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.193+0000 7fc1af40c780 5 client.4426.objecter 1 in flight 2024-02-06T21:44:45.724 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.509+0000 7fc1a9c35640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 29 ==== osd_map(124..124 src has 1..124) v4 ==== 296+0+0 (crc 0 0 0) 0x7fc1a4054800 con 0x55ef778a7460 2024-02-06T21:44:45.724 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.509+0000 7fc1a9c35640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_map(124..124 src has 1..124) v4 2024-02-06T21:44:45.724 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.509+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map got epochs [124,124] > 123 2024-02-06T21:44:45.724 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.509+0000 7fc1a9c35640 3 client.4426.objecter handle_osd_map decoding incremental epoch 124 2024-02-06T21:44:45.724 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.509+0000 7fc1a9c35640 10 client.4426.objecter checking op 20 2024-02-06T21:44:45.724 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.509+0000 7fc1a9c35640 20 client.4426.objecter _calc_target epoch 124 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-02-06T21:44:45.724 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.509+0000 7fc1a9c35640 20 client.4426.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-02-06T21:44:45.724 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.509+0000 7fc1a9c35640 20 client.4426.objecter dump_active .. 0 homeless 2024-02-06T21:44:45.724 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.509+0000 7fc1a9c35640 20 client.4426.objecter 20 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-02-06T21:44:45.724 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.646+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:45.725 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.646+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:45.725 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:44.646+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:45.725 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.646+0000 7fc1a8c33640 10 monclient: tick 2024-02-06T21:44:45.725 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.646+0000 7fc1a8c33640 10 monclient: _check_auth_tickets 2024-02-06T21:44:45.725 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.646+0000 7fc1a8c33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T21:44:45.725 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.706+0000 7fc1ae69e640 10 client.4426.objecter tick 2024-02-06T21:44:45.725 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 <== osd.1 v2:172.21.15.120:6808/3348315891 30 ==== osd_op_reply(20 exist [writefull 0~5] v124'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fc1a404f070 con 0x55ef778a7460 2024-02-06T21:44:45.725 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1acc3b640 10 client.4426.objecter ms_dispatch 0x55ef77852440 osd_op_reply(20 exist [writefull 0~5] v124'7 uv7 ondisk = 0) v8 2024-02-06T21:44:45.725 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1acc3b640 10 client.4426.objecter in handle_osd_op_reply 2024-02-06T21:44:45.725 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1acc3b640 7 client.4426.objecter handle_osd_op_reply 20 ondisk uv 7 in 12.1 attempt 0 2024-02-06T21:44:45.725 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1acc3b640 10 client.4426.objecter op 0 rval 0 len 0 2024-02-06T21:44:45.725 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1acc3b640 15 client.4426.objecter handle_osd_op_reply completed tid 20 2024-02-06T21:44:45.725 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1acc3b640 15 client.4426.objecter _finish_op 20 2024-02-06T21:44:45.725 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1acc3b640 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 4 2024-02-06T21:44:45.725 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1acc3b640 15 client.4426.objecter _session_op_remove 1 20 2024-02-06T21:44:45.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1acc3b640 5 client.4426.objecter 0 in flight 2024-02-06T21:44:45.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1acc3b640 20 librados: complete_aio_write 0x55ef778a71a0 2024-02-06T21:44:45.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (1528 ms) 2024-02-06T21:44:45.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 10 librados: watch_flush enter 2024-02-06T21:44:45.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 10 librados: watch_flush exit 2024-02-06T21:44:45.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 10 client.4426.objecter close_session for osd.1 2024-02-06T21:44:45.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 >> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] conn(0x55ef778a7460 msgr2=0x55ef77a297b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:44:45.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 1 --2- 172.21.15.120:0/2499202281 >> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] conn(0x55ef778a7460 0x55ef77a297b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T21:44:45.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 20 client.4426.objecter put_session s=0x55ef77973370 osd=1 1 2024-02-06T21:44:45.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 20 client.4426.objecter shutdown clearing up homeless session... 2024-02-06T21:44:45.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 10 client.4426.objecter successfully canceled tick 2024-02-06T21:44:45.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc19806b490 msgr2=0x7fc19806d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:44:45.726 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 1 --2- 172.21.15.120:0/2499202281 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc19806b490 0x7fc19806d950 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00c010 tx=0x7fc19c002000 comp rx=0 tx=0).stop 2024-02-06T21:44:45.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 10 monclient: shutdown 2024-02-06T21:44:45.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T21:44:45.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ef778e6b30 msgr2=0x55ef77972e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T21:44:45.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 1 --2- 172.21.15.120:0/2499202281 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ef778e6b30 0x55ef77972e30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000af50 tx=0x7fc1a00087f0 comp rx=0 tx=0).stop 2024-02-06T21:44:45.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1acc3b640 1 -- 172.21.15.120:0/2499202281 reap_dead start 2024-02-06T21:44:45.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 shutdown_connections 2024-02-06T21:44:45.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 >> 172.21.15.120:0/2499202281 conn(0x55ef778a3c10 msgr2=0x55ef7797f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T21:44:45.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 shutdown_connections 2024-02-06T21:44:45.727 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 1 -- 172.21.15.120:0/2499202281 wait complete. 2024-02-06T21:44:45.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.721+0000 7fc1af40c780 1 librados: shutdown 2024-02-06T21:44:45.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [----------] 12 tests from AsioRados (75882 ms total) 2024-02-06T21:44:45.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: 2024-02-06T21:44:45.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [----------] Global test environment tear-down 2024-02-06T21:44:45.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (130137 ms total) 2024-02-06T21:44:45.728 INFO:tasks.workunit.client.0.smithi120.stdout: api_asio: [ PASSED ] 12 tests. 2024-02-06T21:44:47.297 INFO:tasks.ceph.osd.7.smithi135.stderr:2024-02-06T21:44:47.258+0000 ea52640 -1 osd.7 127 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4715.0:2 83.15 83:ad551702:::test:head [create,setxattr key (5) in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) 2024-02-06T21:44:48.293 INFO:tasks.ceph.osd.7.smithi135.stderr:2024-02-06T21:44:48.254+0000 ea52640 -1 osd.7 128 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4715.0:2 83.15 83:ad551702:::test:head [create,setxattr key (5) in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) 2024-02-06T21:44:48.444 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:48.381+0000 12a5a640 -1 osd.1 128 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:5 17.2 17:450225c4:test-rados-api-smithi120-40515-5::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74) 2024-02-06T21:44:49.528 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:49.485+0000 12a5a640 -1 osd.1 129 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:5 17.2 17:450225c4:test-rados-api-smithi120-40515-5::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74) 2024-02-06T21:44:50.086 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:44:50.057+0000 12a5a640 -1 osd.2 128 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4750.0:2 81.0 81.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-02-06T21:44:50.742 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:44:50.624+0000 12a5a640 -1 osd.4 129 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4721.0:2 84.1b 84.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e101) 2024-02-06T21:44:51.230 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:44:51.111+0000 12a5a640 -1 osd.2 129 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4750.0:2 81.0 81.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-02-06T21:44:51.794 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:44:51.714+0000 12a5a640 -1 osd.4 130 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4721.0:2 84.1b 84.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e101) 2024-02-06T21:44:51.946 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:51.936+0000 12a5a640 -1 osd.1 131 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:5 17.2 17:450225c4:test-rados-api-smithi120-40515-5::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74) 2024-02-06T21:44:52.457 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:44:52.360+0000 12a5a640 -1 osd.2 130 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4750.0:2 81.0 81.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-02-06T21:44:52.797 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:44:52.766+0000 12a5a640 -1 osd.4 131 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4721.0:2 84.1b 84.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e101) 2024-02-06T21:44:53.441 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:53.422+0000 12a5a640 -1 osd.1 133 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:5 17.2 17:450225c4:test-rados-api-smithi120-40515-5::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74) 2024-02-06T21:44:54.027 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:44:54.014+0000 12a5a640 -1 osd.4 132 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4721.0:2 84.1b 84.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e101) 2024-02-06T21:44:54.541 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:54.537+0000 12a5a640 -1 osd.1 134 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:5 17.2 17:450225c4:test-rados-api-smithi120-40515-5::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74) 2024-02-06T21:44:55.047 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:44:55.012+0000 12a5a640 -1 osd.2 132 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4750.0:2 81.0 81.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-02-06T21:44:55.057 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:44:55.016+0000 12a5a640 -1 osd.4 133 heartbeat_check: no reply from 172.21.15.120:6812 osd.1 since back 2024-02-06T21:43:35.120886+0000 front 2024-02-06T21:44:54.152505+0000 (oldest deadline 2024-02-06T21:44:53.934094+0000) 2024-02-06T21:44:55.156 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:44:55.146+0000 12a5a640 -1 osd.4 134 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4721.0:2 84.1b 84.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e101) 2024-02-06T21:44:56.082 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:56.056+0000 12a5a640 -1 osd.1 135 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:5 17.2 17:450225c4:test-rados-api-smithi120-40515-5::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74) 2024-02-06T21:44:56.142 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:44:56.130+0000 12a5a640 -1 osd.4 134 heartbeat_check: no reply from 172.21.15.120:6812 osd.1 since back 2024-02-06T21:43:35.120886+0000 front 2024-02-06T21:44:54.152505+0000 (oldest deadline 2024-02-06T21:44:53.934094+0000) 2024-02-06T21:44:56.144 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:44:56.138+0000 12a5a640 -1 osd.4 134 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4721.0:2 84.1b 84.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e101) 2024-02-06T21:44:57.568 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:57.555+0000 12a5a640 -1 osd.1 136 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:5 17.2 17:450225c4:test-rados-api-smithi120-40515-5::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74) 2024-02-06T21:44:57.842 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:44:57.722+0000 12a5a640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.120:6812 osd.1 since back 2024-02-06T21:43:35.120886+0000 front 2024-02-06T21:44:54.152505+0000 (oldest deadline 2024-02-06T21:44:53.934094+0000) 2024-02-06T21:44:58.266 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:44:58.144+0000 12a5a640 -1 osd.4 136 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4721.0:2 84.1b 84.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e101) 2024-02-06T21:44:58.835 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:58.794+0000 12a5a640 -1 osd.1 137 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:5 17.2 17:450225c4:test-rados-api-smithi120-40515-5::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74) 2024-02-06T21:44:59.200 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:44:59.171+0000 12a5a640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.120:6812 osd.1 since back 2024-02-06T21:43:35.120886+0000 front 2024-02-06T21:44:54.152505+0000 (oldest deadline 2024-02-06T21:44:53.934094+0000) 2024-02-06T21:44:59.214 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:44:59.178+0000 12a5a640 -1 osd.4 136 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4721.0:2 84.1b 84.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e101) 2024-02-06T21:45:00.007 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:44:59.938+0000 12a5a640 -1 osd.1 138 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:5 17.2 17:450225c4:test-rados-api-smithi120-40515-5::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74) 2024-02-06T21:45:00.444 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:00.356+0000 12a5a640 -1 osd.4 137 heartbeat_check: no reply from 172.21.15.120:6812 osd.1 since back 2024-02-06T21:43:35.120886+0000 front 2024-02-06T21:44:54.152505+0000 (oldest deadline 2024-02-06T21:44:53.934094+0000) 2024-02-06T21:45:00.447 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:00.363+0000 12a5a640 -1 osd.4 137 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4721.0:2 84.1b 84.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e101) 2024-02-06T21:45:01.107 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:45:01.066+0000 12a5a640 -1 osd.1 139 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:5 17.2 17:450225c4:test-rados-api-smithi120-40515-5::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74) 2024-02-06T21:45:02.135 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:04:41.950 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:02.196 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:04:42.011 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:02.202 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:04:41.967 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:02.207 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:04:41.971 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:02.219 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T21:45:02.207+0000 12a5a640 -1 osd.1 140 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4370.0:5 17.2 17:450225c4:test-rados-api-smithi120-40515-5::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74) 2024-02-06T21:45:02.241 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:02.207+0000 12a5a640 -1 osd.4 139 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4721.0:2 84.1b 84.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e101) 2024-02-06T21:45:02.264 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:04:42.079 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:02.268 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:04:42.083 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:02.403 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:04:42.167 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:02.410 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:04:42.225 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:02.504 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:04:42.319 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:03.393 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:03.383+0000 12a5a640 -1 osd.4 139 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4721.0:2 84.1b 84.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e101) 2024-02-06T21:45:05.306 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:05.206+0000 12a5a640 -1 osd.4 142 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4721.0:2 84.1b 84:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) 2024-02-06T21:45:06.965 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:06.926+0000 12a5a640 -1 osd.4 143 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 88.6 88.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e114) 2024-02-06T21:45:08.429 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:08.256+0000 12a5a640 -1 osd.4 145 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 88.6 88.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e114) 2024-02-06T21:45:10.519 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:10.492+0000 12a5a640 -1 osd.4 146 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 88.6 88.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e114) 2024-02-06T21:45:12.269 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:12.195+0000 12a5a640 -1 osd.4 147 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 88.6 88:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) 2024-02-06T21:45:13.289 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:13.237+0000 12a5a640 -1 osd.4 148 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 88.6 88:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) 2024-02-06T21:45:14.261 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:14.253+0000 12a5a640 -1 osd.4 150 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 88.6 88:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) 2024-02-06T21:45:15.523 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:15.436+0000 12a5a640 -1 osd.4 153 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 88.6 88:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) 2024-02-06T21:45:16.785 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:45:16.727+0000 12a5a640 -1 osd.2 149 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4750.0:2 81.0 81:0384ab8e:::oid:head [create] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e96) 2024-02-06T21:45:17.109 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:17.049+0000 12a5a640 -1 osd.4 153 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 88.6 88:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) 2024-02-06T21:45:18.186 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:18.119+0000 12a5a640 -1 osd.4 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 88.6 88:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) 2024-02-06T21:45:19.345 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:19.183+0000 12a5a640 -1 osd.4 155 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 88.6 88:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) 2024-02-06T21:45:20.372 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:20.349+0000 12a5a640 -1 osd.4 156 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 88.6 88:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) 2024-02-06T21:45:22.010 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T21:45:21.986+0000 12a5a640 -1 osd.4 158 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 88.6 88:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) 2024-02-06T21:45:22.441 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:05:02.255 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:22.498 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:05:02.313 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:22.501 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:05:02.315 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:22.503 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:02.268 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:22.509 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:02.273 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:22.579 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:05:02.393 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:22.596 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:02.360 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:22.619 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:05:02.434 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:22.705 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:05:02.520 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:39.021 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: Running main() from gmock_main.cc 2024-02-06T21:45:39.021 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-02-06T21:45:39.021 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [----------] Global test environment set-up. 2024-02-06T21:45:39.021 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-02-06T21:45:39.021 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-02-06T21:45:39.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ OK ] LibRadosStat.Stat (40309 ms) 2024-02-06T21:45:39.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-02-06T21:45:39.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (9734 ms) 2024-02-06T21:45:39.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-02-06T21:45:39.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ OK ] LibRadosStat.StatNS (20579 ms) 2024-02-06T21:45:39.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-02-06T21:45:39.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (17 ms) 2024-02-06T21:45:39.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-02-06T21:45:39.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (1679 ms) 2024-02-06T21:45:39.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [----------] 5 tests from LibRadosStat (72318 ms total) 2024-02-06T21:45:39.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: 2024-02-06T21:45:39.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-02-06T21:45:39.022 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-02-06T21:45:39.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (54652 ms) 2024-02-06T21:45:39.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-02-06T21:45:39.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (34276 ms) 2024-02-06T21:45:39.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-02-06T21:45:39.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (2 ms) 2024-02-06T21:45:39.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-02-06T21:45:39.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (9748 ms) 2024-02-06T21:45:39.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (98678 ms total) 2024-02-06T21:45:39.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: 2024-02-06T21:45:39.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [----------] Global test environment tear-down 2024-02-06T21:45:39.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (183357 ms total) 2024-02-06T21:45:39.023 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat: [ PASSED ] 9 tests. 2024-02-06T21:45:42.663 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:05:22.478 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:42.666 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:05:22.481 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:42.722 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:22.485 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:42.726 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:22.491 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:42.751 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:05:22.565 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:42.753 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:05:22.567 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:42.796 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:05:22.610 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:42.848 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:22.612 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:45:42.867 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:05:22.682 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:46:10.254 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:05:50.068 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:46:10.257 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:05:50.070 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:46:10.717 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:05:50.532 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:46:10.758 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:50.522 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:46:10.788 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:50.552 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:46:10.988 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:50.752 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:46:11.337 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:05:51.149 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:46:11.337 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:05:51.150 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:46:11.581 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:05:51.396 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:46:21.301 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-02-06T21:46:21.301 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-02-06T21:46:21.302 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-02-06T21:46:21.302 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-02-06T21:46:21.302 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-02-06T21:46:21.302 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (598 ms) 2024-02-06T21:46:21.302 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-02-06T21:46:21.302 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (587 ms) 2024-02-06T21:46:21.302 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-02-06T21:46:21.302 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (224387 ms) 2024-02-06T21:46:21.302 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (225572 ms total) 2024-02-06T21:46:21.302 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd_pp: 2024-02-06T21:46:21.302 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-02-06T21:46:21.302 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (225572 ms total) 2024-02-06T21:46:21.302 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-02-06T21:46:21.667 INFO:tasks.workunit.client.0.smithi120.stdout: cmd: Running main() from gmock_main.cc 2024-02-06T21:46:21.667 INFO:tasks.workunit.client.0.smithi120.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-02-06T21:46:21.667 INFO:tasks.workunit.client.0.smithi120.stdout: cmd: [----------] Global test environment set-up. 2024-02-06T21:46:21.667 INFO:tasks.workunit.client.0.smithi120.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-02-06T21:46:21.668 INFO:tasks.workunit.client.0.smithi120.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-02-06T21:46:21.668 INFO:tasks.workunit.client.0.smithi120.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (5975 ms) 2024-02-06T21:46:21.668 INFO:tasks.workunit.client.0.smithi120.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-02-06T21:46:21.668 INFO:tasks.workunit.client.0.smithi120.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (29329 ms) 2024-02-06T21:46:21.668 INFO:tasks.workunit.client.0.smithi120.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-02-06T21:46:21.668 INFO:tasks.workunit.client.0.smithi120.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (190437 ms) 2024-02-06T21:46:21.668 INFO:tasks.workunit.client.0.smithi120.stdout: cmd: [----------] 3 tests from NeoRadosCmd (225741 ms total) 2024-02-06T21:46:21.668 INFO:tasks.workunit.client.0.smithi120.stdout: cmd: 2024-02-06T21:46:21.669 INFO:tasks.workunit.client.0.smithi120.stdout: cmd: [----------] Global test environment tear-down 2024-02-06T21:46:21.669 INFO:tasks.workunit.client.0.smithi120.stdout: cmd: [==========] 3 tests from 1 test suite ran. (225741 ms total) 2024-02-06T21:46:21.669 INFO:tasks.workunit.client.0.smithi120.stdout: cmd: [ PASSED ] 3 tests. 2024-02-06T21:46:25.844 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-02-06T21:46:25.844 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-02-06T21:46:25.844 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-02-06T21:46:25.844 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-02-06T21:46:25.844 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-02-06T21:46:25.844 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-02-06T21:46:25.844 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-02-06T21:46:25.844 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (42942 ms) 2024-02-06T21:46:25.844 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-02-06T21:46:25.844 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (2564 ms) 2024-02-06T21:46:25.844 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-02-06T21:46:25.844 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (7231 ms) 2024-02-06T21:46:25.845 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-02-06T21:46:25.845 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (70600 ms) 2024-02-06T21:46:25.845 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-02-06T21:46:25.845 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (6111 ms) 2024-02-06T21:46:25.845 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-02-06T21:46:25.845 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (9049 ms) 2024-02-06T21:46:25.845 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-02-06T21:46:25.845 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (14718 ms) 2024-02-06T21:46:25.845 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-02-06T21:46:25.845 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (6349 ms) 2024-02-06T21:46:25.845 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-02-06T21:46:25.845 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (17150 ms) 2024-02-06T21:46:25.845 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-02-06T21:46:25.845 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (7692 ms) 2024-02-06T21:46:25.845 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-02-06T21:46:25.846 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (25623 ms) 2024-02-06T21:46:25.846 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-02-06T21:46:25.846 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (747 ms) 2024-02-06T21:46:25.846 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-02-06T21:46:25.846 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (6601 ms) 2024-02-06T21:46:25.846 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-02-06T21:46:25.846 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (1065 ms) 2024-02-06T21:46:25.846 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-02-06T21:46:25.846 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (273 ms) 2024-02-06T21:46:25.846 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-02-06T21:46:25.846 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (117 ms) 2024-02-06T21:46:25.846 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (218832 ms total) 2024-02-06T21:46:25.846 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: 2024-02-06T21:46:25.846 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-02-06T21:46:25.846 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (230039 ms total) 2024-02-06T21:46:25.847 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-02-06T21:46:26.825 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-02-06T21:46:26.825 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-02-06T21:46:26.825 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-02-06T21:46:26.825 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-02-06T21:46:26.825 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: seed 40868 2024-02-06T21:46:26.825 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-02-06T21:46:26.825 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (40741 ms) 2024-02-06T21:46:26.825 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-02-06T21:46:26.825 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (56964 ms) 2024-02-06T21:46:26.825 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-02-06T21:46:26.826 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (10 ms) 2024-02-06T21:46:26.826 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-02-06T21:46:26.826 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (24609 ms) 2024-02-06T21:46:26.826 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-02-06T21:46:26.826 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (10248 ms) 2024-02-06T21:46:26.826 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (132572 ms total) 2024-02-06T21:46:26.826 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: 2024-02-06T21:46:26.826 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-02-06T21:46:26.826 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-02-06T21:46:26.826 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (19858 ms) 2024-02-06T21:46:26.826 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-02-06T21:46:26.826 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (24 ms) 2024-02-06T21:46:26.826 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-02-06T21:46:26.826 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (5046 ms) 2024-02-06T21:46:26.826 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-02-06T21:46:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (42191 ms) 2024-02-06T21:46:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (67119 ms total) 2024-02-06T21:46:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: 2024-02-06T21:46:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-02-06T21:46:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (231134 ms total) 2024-02-06T21:46:26.827 INFO:tasks.workunit.client.0.smithi120.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-02-06T21:46:38.800 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: Running main() from gmock_main.cc 2024-02-06T21:46:38.800 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-02-06T21:46:38.800 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: [----------] Global test environment set-up. 2024-02-06T21:46:38.800 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-02-06T21:46:38.800 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-02-06T21:46:38.800 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (953 ms) 2024-02-06T21:46:38.800 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-02-06T21:46:38.800 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (200 ms) 2024-02-06T21:46:38.800 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-02-06T21:46:38.801 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (229675 ms) 2024-02-06T21:46:38.801 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-02-06T21:46:38.801 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:18.748079+0000 mon.b [INF] from='client.? 172.21.15.120:0/290870531' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi120-40702-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-06T21:46:38.801 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:19.440721+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on back (longest 34315.561ms) (OSD_SLOW_PING_TIME_BACK) 2024-02-06T21:46:38.801 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:19.442052+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on front (longest 28093.594ms) (OSD_SLOW_PING_TIME_FRONT) 2024-02-06T21:46:38.801 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:19.442554+0000 mon.a [WRN] Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-06T21:46:38.801 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:19.891152+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi120-40702-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-06T21:46:38.801 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:24.386366+0000 mon.a [INF] from='client.? 172.21.15.120:0/2402338340' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi120-40868-7"}]': finished 2024-02-06T21:46:38.801 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:24.388902+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi120-40702-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-06T21:46:38.801 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:25.113649+0000 mon.a [INF] from='client.? 172.21.15.120:0/3988097999' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi120-40515-16"}]: dispatch 2024-02-06T21:46:38.801 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:25.270802+0000 mon.a [INF] from='client.? 172.21.15.120:0/3988097999' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi120-40515-16"}]: dispatch 2024-02-06T21:46:38.801 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:25.679868+0000 mon.a [INF] from='client.? 172.21.15.120:0/3988097999' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi120-40515-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-06T21:46:38.801 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:26.873918+0000 mon.c [INF] from='client.? 172.21.15.120:0/3346266462' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi120-41645-2"}]: dispatch 2024-02-06T21:46:38.802 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:26.972941+0000 mon.a [WRN] Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-06T21:46:38.802 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:28.175422+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi120-41645-2"}]: dispatch 2024-02-06T21:46:38.802 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:28.688867+0000 mon.a [INF] from='client.? 172.21.15.120:0/3982381297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi120-40610-10"}]: dispatch 2024-02-06T21:46:39.555 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:29.119078+0000 mon.c [INF] from='client.? 172.21.15.120:0/169898 api_list: [==========] Running 11 tests from 3 test suites. 2024-02-06T21:46:39.555 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [----------] Global test environment set-up. 2024-02-06T21:46:39.555 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [----------] 7 tests from LibRadosList 2024-02-06T21:46:39.555 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-02-06T21:46:39.556 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ OK ] LibRadosList.ListObjects (212088 ms) 2024-02-06T21:46:39.556 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-02-06T21:46:39.556 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (2965 ms) 2024-02-06T21:46:39.556 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-02-06T21:46:39.556 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: myset foo1,foo2,foo3 2024-02-06T21:46:39.556 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo1 2024-02-06T21:46:39.556 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo2 2024-02-06T21:46:39.556 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo3 2024-02-06T21:46:39.556 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: myset foo1,foo4,foo5 2024-02-06T21:46:39.556 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo4 2024-02-06T21:46:39.556 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo5 2024-02-06T21:46:39.556 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo1 2024-02-06T21:46:39.556 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: myset foo6,foo7 2024-02-06T21:46:39.556 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo7 2024-02-06T21:46:39.557 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo6 2024-02-06T21:46:39.557 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-02-06T21:46:39.557 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: ns1:foo4 2024-02-06T21:46:39.557 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: ns1:foo5 2024-02-06T21:46:39.557 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: ns2:foo7 2024-02-06T21:46:39.557 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: ns2:foo6 2024-02-06T21:46:39.557 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: ns1:foo1 2024-02-06T21:46:39.557 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: :foo1 2024-02-06T21:46:39.557 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: :foo2 2024-02-06T21:46:39.557 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: :foo3 2024-02-06T21:46:39.557 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (10900 ms) 2024-02-06T21:46:39.557 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-02-06T21:46:39.557 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 1 0 2024-02-06T21:46:39.557 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 10 0 2024-02-06T21:46:39.557 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 13 0 2024-02-06T21:46:39.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 7 0 2024-02-06T21:46:39.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 14 0 2024-02-06T21:46:39.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 0 0 2024-02-06T21:46:39.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 15 0 2024-02-06T21:46:39.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 11 0 2024-02-06T21:46:39.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 5 0 2024-02-06T21:46:39.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 8 0 2024-02-06T21:46:39.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 6 0 2024-02-06T21:46:39.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 3 0 2024-02-06T21:46:39.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 4 0 2024-02-06T21:46:39.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 12 0 2024-02-06T21:46:39.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 9 0 2024-02-06T21:46:39.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 2 0 2024-02-06T21:46:39.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-02-06T21:46:39.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (7190 ms) 2024-02-06T21:46:39.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-02-06T21:46:39.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: x cursor=MIN 2024-02-06T21:46:39.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=1 cursor=13:02547ec2:::1:head 2024-02-06T21:46:39.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=10 cursor=13:52ea6a34:::10:head 2024-02-06T21:46:39.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=13 cursor=13:566253c9:::13:head 2024-02-06T21:46:39.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=7 cursor=13:5c6b0b28:::7:head 2024-02-06T21:46:39.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=14 cursor=13:62a1935d:::14:head 2024-02-06T21:46:39.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=0 cursor=13:6cac518f:::0:head 2024-02-06T21:46:39.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=15 cursor=13:863748b0:::15:head 2024-02-06T21:46:39.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=11 cursor=13:89d3ae78:::11:head 2024-02-06T21:46:39.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=5 cursor=13:b29083e3:::5:head 2024-02-06T21:46:39.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=8 cursor=13:bd63b0f1:::8:head 2024-02-06T21:46:39.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=6 cursor=13:c4fdafeb:::6:head 2024-02-06T21:46:39.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=3 cursor=13:cfc208b3:::3:head 2024-02-06T21:46:39.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=4 cursor=13:d83876eb:::4:head 2024-02-06T21:46:39.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=12 cursor=13:de5d7c5f:::12:head 2024-02-06T21:46:39.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=9 cursor=13:e960b815:::9:head 2024-02-06T21:46:39.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > oid=2 cursor=13:f905c69b:::2:head 2024-02-06T21:46:39.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: FIRST> seek to MIN oid=1 2024-02-06T21:46:39.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : oid=1 cursor=13:02547ec2:::1:head 2024-02-06T21:46:39.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:02547ec2:::1:head 2024-02-06T21:46:39.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:02547ec2:::1:head -> 1 2024-02-06T21:46:39.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : oid=10 cursor=13:52ea6a34:::10:head 2024-02-06T21:46:39.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:52ea6a34:::10:head 2024-02-06T21:46:39.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2024-02-06T21:46:39.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : oid=13 cursor=13:566253c9:::13:head 2024-02-06T21:46:39.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:566253c9:::13:head 2024-02-06T21:46:39.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:566253c9:::13:head -> 13 2024-02-06T21:46:39.561 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : oid=7 cursor=13:5c6b0b28:::7:head 2024-02-06T21:46:39.561 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2024-02-06T21:46:39.561 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2024-02-06T21:46:40.987 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : api_lock_pp: Running main() from gmock_main.cc 2024-02-06T21:46:40.987 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-02-06T21:46:40.987 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-02-06T21:46:40.987 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-02-06T21:46:40.987 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: seed 40610 2024-02-06T21:46:40.987 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-02-06T21:46:40.988 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (50655 ms) 2024-02-06T21:46:40.988 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-02-06T21:46:40.988 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (19944 ms) 2024-02-06T21:46:40.988 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-02-06T21:46:40.988 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (43951 ms) 2024-02-06T21:46:40.988 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-02-06T21:46:40.988 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (3342 ms) 2024-02-06T21:46:40.988 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-02-06T21:46:40.989 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (12624 ms) 2024-02-06T21:46:40.989 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-02-06T21:46:40.989 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (3989 ms) 2024-02-06T21:46:40.989 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-02-06T21:46:40.989 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (244 ms) 2024-02-06T21:46:40.989 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-02-06T21:46:40.989 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (39352 ms) 2024-02-06T21:46:40.989 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (174102 ms total) 2024-02-06T21:46:40.989 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: 2024-02-06T21:46:40.989 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-02-06T21:46:40.989 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-02-06T21:46:40.989 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (37148 ms) 2024-02-06T21:46:40.989 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-02-06T21:46:40.989 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (906 ms) 2024-02-06T21:46:40.989 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-02-06T21:46:40.990 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (2552 ms) 2024-02-06T21:46:40.990 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-02-06T21:46:40.990 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1100 ms) 2024-02-06T21:46:40.990 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-02-06T21:46:40.990 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (295 ms) 2024-02-06T21:46:40.990 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-02-06T21:46:40.990 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (223 ms) 2024-02-06T21:46:40.990 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-02-06T21:46:40.990 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (212 ms) 2024-02-06T21:46:40.990 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-02-06T21:46:40.990 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (2578 ms) 2024-02-06T21:46:40.990 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (45014 ms total) 2024-02-06T21:46:40.990 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: 2024-02-06T21:46:40.990 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-02-06T21:46:40.991 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (245422 ms total) 2024-02-06T21:46:40.991 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-02-06T21:46:43.285 INFO:tasks.workunit.client.0.smithi120.stdout:oid=14 cursor=13:62a1935d:::14:head 2024-02-06T21:46:43.285 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:62a1935d:::14:head 2024-02-06T21:46:43.285 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:62a1935d:::14:head -> 14 2024-02-06T21:46:43.285 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : oid=0 cursor=13:6cac518f:::0:head 2024-02-06T21:46:43.285 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:6cac518f:::0:head 2024-02-06T21:46:43.285 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:6cac518f:::0:head -> 0 2024-02-06T21:46:43.286 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : oid=15 cursor=13:863748b0:::15:head 2024-02-06T21:46:43.286 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:863748b0:::15:head 2024-02-06T21:46:43.286 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:863748b0:::15:head -> 15 2024-02-06T21:46:43.286 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : oid=11 cursor=13:89d3ae78:::11:head 2024-02-06T21:46:43.286 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:89d3ae78:::11:head 2024-02-06T21:46:43.286 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2024-02-06T21:46:43.286 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : oid=5 cursor=13:b29083e3:::5:head 2024-02-06T21:46:43.286 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:b29083e3:::5:head 2024-02-06T21:46:43.286 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:b29083e3:::5:head -> 5 2024-02-06T21:46:43.286 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : oid=8 cursor=13:bd63b0f1:::8:head 2024-02-06T21:46:43.286 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2024-02-06T21:46:43.286 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2024-02-06T21:46:43.286 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : oid=6 cursor=13:c4fdafeb:::6:head 2024-02-06T21:46:43.286 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2024-02-06T21:46:43.286 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2024-02-06T21:46:43.287 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : oid=3 cursor=13:cfc208b3:::3:head 2024-02-06T21:46:43.287 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:cfc208b3:::3:head 2024-02-06T21:46:43.287 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2024-02-06T21:46:43.287 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : oid=4 cursor=13:d83876eb:::4:head 2024-02-06T21:46:43.287 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:d83876eb:::4:head 2024-02-06T21:46:43.287 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:d83876eb:::4:head -> 4 2024-02-06T21:46:43.287 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : oid=12 cursor=13:de5d7c5f:::12:head 2024-02-06T21:46:43.287 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2024-02-06T21:46:43.287 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2024-02-06T21:46:43.287 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : oid=9 cursor=13:e960b815:::9:head 2024-02-06T21:46:43.287 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:e960b815:::9:head 2024-02-06T21:46:43.287 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:e960b815:::9:head -> 9 2024-02-06T21:46:43.287 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : oid=2 cursor=13:f905c69b:::2:head 2024-02-06T21:46:43.287 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:f905c69b:::2:head 2024-02-06T21:46:43.287 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:f905c69b:::2:head -> 2 2024-02-06T21:46:43.288 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:566253c9:::13:head 2024-02-06T21:46:43.288 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:566253c9:::13:head expected=13:566253c9:::13:head 2024-02-06T21:46:43.288 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:566253c9:::13:head -> 13 2024-02-06T21:46:43.288 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : entry=13 expected=13 2024-02-06T21:46:43.288 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2024-02-06T21:46:43.288 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:5c6b0b28:::7:head expected=13:5c6b0b28:::7:head 2024-02-06T21:46:43.288 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2024-02-06T21:46:43.288 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : entry=7 expected=7 2024-02-06T21:46:43.288 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:52ea6a34:::10:head 2024-02-06T21:46:43.288 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:52ea6a34:::10:head expected=13:52ea6a34:::10:head 2024-02-06T21:46:43.288 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2024-02-06T21:46:43.288 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : entry=10 expected=10 2024-02-06T21:46:43.288 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:d83876eb:::4:head 2024-02-06T21:46:43.288 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:d83876eb:::4:head expected=13:d83876eb:::4:head 2024-02-06T21:46:43.289 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:d83876eb:::4:head -> 4 2024-02-06T21:46:43.289 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : entry=4 expected=4 2024-02-06T21:46:43.289 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:f905c69b:::2:head 2024-02-06T21:46:43.289 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:f905c69b:::2:head expected=13:f905c69b:::2:head 2024-02-06T21:46:43.289 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:f905c69b:::2:head -> 2 2024-02-06T21:46:43.289 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : entry=2 expected=2 2024-02-06T21:46:43.289 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2024-02-06T21:46:43.289 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:de5d7c5f:::12:head expected=13:de5d7c5f:::12:head 2024-02-06T21:46:43.289 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2024-02-06T21:46:43.289 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : entry=12 expected=12 2024-02-06T21:46:43.289 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:e960b815:::9:head 2024-02-06T21:46:43.289 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:e960b815:::9:head expected=13:e960b815:::9:head 2024-02-06T21:46:43.289 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:e960b815:::9:head -> 9 2024-02-06T21:46:43.289 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : entry=9 expected=9 2024-02-06T21:46:43.289 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:cfc208b3:::3:head 2024-02-06T21:46:43.290 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:cfc208b3:::3:head expected=13:cfc208b3:::3:head 2024-02-06T21:46:43.290 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2024-02-06T21:46:53.530 INFO:tasks.workunit.client.0.smithi120.stdout: api api_lock: Running main() from gmock_main.cc 2024-02-06T21:46:53.530 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-02-06T21:46:53.530 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [----------] Global test environment set-up. 2024-02-06T21:46:53.530 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-02-06T21:46:53.530 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-02-06T21:46:53.530 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (11334 ms) 2024-02-06T21:46:53.530 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-02-06T21:46:53.530 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLock.LockShared (7737 ms) 2024-02-06T21:46:53.530 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-02-06T21:46:53.530 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (58973 ms) 2024-02-06T21:46:53.530 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-02-06T21:46:53.531 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (4283 ms) 2024-02-06T21:46:53.531 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-02-06T21:46:53.531 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (47152 ms) 2024-02-06T21:46:53.531 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-02-06T21:46:53.531 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLock.Unlock (5775 ms) 2024-02-06T21:46:53.531 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-02-06T21:46:53.531 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (9058 ms) 2024-02-06T21:46:53.531 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-02-06T21:46:53.531 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (29458 ms) 2024-02-06T21:46:53.531 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [----------] 8 tests from LibRadosLock (173770 ms total) 2024-02-06T21:46:53.531 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: 2024-02-06T21:46:53.531 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-02-06T21:46:53.531 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-02-06T21:46:53.531 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (13476 ms) 2024-02-06T21:46:53.532 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-02-06T21:46:53.532 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (21985 ms) 2024-02-06T21:46:53.532 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-02-06T21:46:53.532 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1132 ms) 2024-02-06T21:46:53.532 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-02-06T21:46:53.532 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1172 ms) 2024-02-06T21:46:53.532 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-02-06T21:46:53.532 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (1449 ms) 2024-02-06T21:46:53.532 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-02-06T21:46:53.532 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (657 ms) 2024-02-06T21:46:53.532 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-02-06T21:46:53.532 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (368 ms) 2024-02-06T21:46:53.532 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-02-06T21:46:53.532 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (223 ms) 2024-02-06T21:46:53.532 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (40462 ms total) 2024-02-06T21:46:53.533 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: 2024-02-06T21:46:53.533 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [----------] Global test environment tear-down 2024-02-06T21:46:53.533 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (257963 ms total) 2024-02-06T21:46:53.533 INFO:tasks.workunit.client.0.smithi120.stdout: api_lock: [ PASSED ] 16 tests. 2024-02-06T21:46:54.183 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_1_[41329]: starting. 2024-02-06T21:46:54.183 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_1_[41329]: creating pool ceph_test_rados_list_parallel.smithi120-41274 2024-02-06T21:46:54.184 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_1_[41329]: created object 0... 2024-02-06T21:46:54.184 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_1_[41329]: created object 25... 2024-02-06T21:46:54.184 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_1_[41329]: created object 49... 2024-02-06T21:46:54.184 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_1_[41329]: finishing. 2024-02-06T21:46:54.184 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_1_[41329]: shutting down. 2024-02-06T21:46:54.184 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_2_[41330]: starting. 2024-02-06T21:46:54.184 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_2_[41330]: listing objects. 2024-02-06T21:46:54.184 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_2_[41330]: listed object 0... 2024-02-06T21:46:54.184 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_2_[41330]: listed object 25... 2024-02-06T21:46:54.184 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_2_[41330]: saw 50 objects 2024-02-06T21:46:54.184 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_2_[41330]: shutting down. 2024-02-06T21:46:54.184 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:46:54.184 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_3_[67641]: starting. 2024-02-06T21:46:54.184 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_3_[67641]: creating pool ceph_test_rados_list_parallel.smithi120-41274 2024-02-06T21:46:54.185 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_3_[67641]: created object 0... 2024-02-06T21:46:54.185 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_3_[67641]: created object 25... 2024-02-06T21:46:54.185 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_3_[67641]: created object 49... 2024-02-06T21:46:54.185 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_3_[67641]: finishing. 2024-02-06T21:46:54.185 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_3_[67641]: shutting down. 2024-02-06T21:46:54.185 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:46:54.185 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_4_[67642]: starting. 2024-02-06T21:46:54.185 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_4_[67642]: listing objects. 2024-02-06T21:46:54.185 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_4_[67642]: listed object 0... 2024-02-06T21:46:54.185 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_4_[67642]: listed object 25... 2024-02-06T21:46:54.185 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_4_[67642]: saw 43 objects 2024-02-06T21:46:54.185 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_4_[67642]: shutting down. 2024-02-06T21:46:54.185 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:46:54.185 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_5_[67643]: starting. 2024-02-06T21:46:54.185 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_5_[67643]: removed 25 objects... 2024-02-06T21:46:54.186 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_5_[67643]: removed half of the objects 2024-02-06T21:46:54.186 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_5_[67643]: removed 50 objects... 2024-02-06T21:46:54.186 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_5_[67643]: removed 50 objects 2024-02-06T21:46:54.186 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_5_[67643]: shutting down. 2024-02-06T21:46:54.186 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:46:54.186 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:46:54.186 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_6_[78840]: starting. 2024-02-06T21:46:54.186 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_6_[78840]: creating pool ceph_test_rados_list_parallel.smithi120-41274 2024-02-06T21:46:54.186 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_6_[78840]: created object 0... 2024-02-06T21:46:54.186 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_6_[78840]: created object 25... 2024-02-06T21:46:54.186 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_6_[78840]: created object 49... 2024-02-06T21:46:54.186 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_6_[78840]: finishing. 2024-02-06T21:46:54.186 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_6_[78840]: shutting down. 2024-02-06T21:46:54.186 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:46:54.186 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:46:54.187 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_7_[78841]: starting. 2024-02-06T21:46:54.187 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_7_[78841]: listing objects. 2024-02-06T21:46:54.187 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_7_[78841]: listed object 0... 2024-02-06T21:46:54.187 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_7_[78841]: listed object 25... 2024-02-06T21:46:54.187 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_7_[78841]: listed object 50... 2024-02-06T21:46:54.187 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_7_[78841]: saw 53 objects 2024-02-06T21:46:54.187 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_7_[78841]: shutting down. 2024-02-06T21:46:54.187 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:46:54.187 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:46:54.187 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_8_[78842]: starting. 2024-02-06T21:46:54.187 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_8_[78842]: added 25 objects... 2024-02-06T21:46:54.187 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_8_[78842]: added half of the objects 2024-02-06T21:46:54.187 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_8_[78842]: added 50 objects... 2024-02-06T21:46:54.187 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_8_[78842]: added 50 objects 2024-02-06T21:46:54.187 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_8_[78842]: shutting down. 2024-02-06T21:46:54.188 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:46:54.188 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:46:55.805 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: **********************9183' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-06T21:46:55.805 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:29.585392+0000 mon.b [INF] from='client.? 172.21.15.120:0/1575869528' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi120-40567-10"}]: dispatch 2024-02-06T21:46:55.805 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:29.844166+0000 mon.c [INF] from='client.? 172.21.15.120:0/3966011274' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi120-40537-22"}]: dispatch 2024-02-06T21:46:55.805 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:31.337575+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi120-40567-10"}]: dispatch 2024-02-06T21:46:55.805 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:31.426627+0000 client.admin [INF] onexx 2024-02-06T21:46:55.805 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:31.461615+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-06T21:46:55.806 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:31.479929+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi120-40537-22"}]: dispatch 2024-02-06T21:46:55.806 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:31.492568+0000 mon.a [INF] from='client.? 172.21.15.120:0/3988097999' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi120-40515-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-06T21:46:55.806 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:32.040391+0000 mon.a [INF] from='client.? 172.21.15.120:0/3988097999' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi120-40515-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi120-40515-16"}]: dispatch 2024-02-06T21:46:55.806 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:36.901098+0000 mon.c [INF] from='client.? 172.21.15.120:0/3346266462' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi120-41645-2"}]: dispatch 2024-02-06T21:46:55.806 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:37.059930+0000 mon.c [INF] from='client.? 172.21.15.120:0/3966011274' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi120-40537-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-06T21:46:55.806 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:37.277479+0000 mon.b [INF] from='client.? 172.21.15.120:0/1575869528' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi120-40567-10"}]: dispatch 2024-02-06T21:46:55.806 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:37.867635+0000 mon.b [INF] from='client.? 172.21.15.120:0/290870531' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi120-40702-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-06T21:46:55.806 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:38.926022+0000 mon.c [INF] from='client.? 172.21.15.120:0/1698989183' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-06T21:46:55.806 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:39.361577+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi120-40567-10"}]: dispatch 2024-02-06T21:46:55.806 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:39.374725+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi120-40702-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-06T21:46:55.806 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:39.809498+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi120-41645-2"}]: dispatch 2024-02-06T21:47:01.527 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:39.823288+0000 mon.a [INF] from='client.? 172.21.15.120:0/3988097999' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi120-40515-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi120-40515-16"}]': finished 2024-02-06T21:47:01.528 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:39.825391+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi120-40471-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-06T21:47:01.528 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:39.827754+0000 mon.a [INF] from='client.? 172.21.15.120:0/3982381297' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi120-40610-10"}]': finished 2024-02-06T21:47:01.528 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:40.082038+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi120-40537-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-06T21:47:01.528 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:41.427050+0000 client.admin [INF] twoxx 2024-02-06T21:47:01.528 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:41.471963+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-06T21:47:01.528 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:49.386083+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi120-40567-10"}]': finished 2024-02-06T21:47:01.528 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:49.388806+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi120-40702-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-06T21:47:01.528 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:49.391732+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi120-41645-2"}]': finished 2024-02-06T21:47:01.528 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:49.394510+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi120-40537-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-06T21:47:01.528 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:52.069224+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-06T21:47:01.528 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:51.436096+0000 mon.c [INF] from='client.? 172.21.15.120:0/3966011274' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi120-40537-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi120-40537-22"}]: dispatch 2024-02-06T21:47:01.529 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:53.036053+0000 mon.a [WRN] Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-06T21:47:01.529 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:53.777012+0000 mon.c [INF] from='client.? 172.21.15.120:0/3306111977' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi120-41645-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-06T21:47:01.529 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:54.005433+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi120-40537-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi120-40537-22"}]: dispatch 2024-02-06T21:47:01.529 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:54.108748+0000 mon.b [INF] from='client.? 172.21.15.120:0/290870531' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi120-40702-6", "tierpool": "test-rados-api-smithi120-40702-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-06T21:47:01.529 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:55.551075+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi120-40702-6", "tierpool": "test-rados-api-smithi120-40702-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-06T21:47:11.008 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:06:50.822 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:11.015 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:06:50.826 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:11.238 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:06:51.053 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:11.272 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:06:51.036 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:11.281 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:06:51.045 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:11.486 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:06:51.249 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:11.633 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:06:51.447 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:11.635 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:06:51.450 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:11.870 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:06:51.684 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:12.613 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_1_[41283]: starting. 2024-02-06T21:47:12.614 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_1_[41283]: creating pool ceph_test_rados_open_pools_parallel.smithi120-41221 2024-02-06T21:47:12.614 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_1_[41283]: created object 0... 2024-02-06T21:47:12.614 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_1_[41283]: created object 25... 2024-02-06T21:47:12.614 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_1_[41283]: created object 49... 2024-02-06T21:47:12.614 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_1_[41283]: finishing. 2024-02-06T21:47:12.614 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_1_[41283]: shutting down. 2024-02-06T21:47:12.614 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_2_[41284]: starting. 2024-02-06T21:47:12.614 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_2_[41284]: rados_pool_create. 2024-02-06T21:47:12.614 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_2_[41284]: rados_ioctx_create. 2024-02-06T21:47:12.614 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_2_[41284]: shutting down. 2024-02-06T21:47:12.614 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: ******************************* 2024-02-06T21:47:12.614 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_3_[68920]: starting. 2024-02-06T21:47:12.614 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_3_[68920]: creating pool ceph_test_rados_open_pools_parallel.smithi120-41221 2024-02-06T21:47:12.614 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_3_[68920]: created object 0... 2024-02-06T21:47:12.615 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_3_[68920]: created object 25... 2024-02-06T21:47:12.615 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_3_[68920]: created object 49... 2024-02-06T21:47:12.615 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_3_[68920]: finishing. 2024-02-06T21:47:12.615 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_3_[68920]: shutting down. 2024-02-06T21:47:12.615 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: ******************************* 2024-02-06T21:47:12.615 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_4_[68921]: starting. 2024-02-06T21:47:12.615 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_4_[68921]: rados_pool_create. 2024-02-06T21:47:12.615 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_4_[68921]: rados_ioctx_create. 2024-02-06T21:47:12.615 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: process_4_[68921]: shutting down. 2024-02-06T21:47:12.615 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: ******************************* 2024-02-06T21:47:12.615 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: ******************************* 2024-02-06T21:47:12.615 INFO:tasks.workunit.client.0.smithi120.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-02-06T21:47:13.708 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_1_[41312]: starting. 2024-02-06T21:47:13.708 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_1_[41312]: creating pool ceph_test_rados_delete_pools_parallel.smithi120-41265 2024-02-06T21:47:13.708 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_1_[41312]: created object 0... 2024-02-06T21:47:13.708 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_1_[41312]: created object 25... 2024-02-06T21:47:13.708 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_1_[41312]: created object 49... 2024-02-06T21:47:13.708 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_1_[41312]: finishing. 2024-02-06T21:47:13.708 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_1_[41312]: shutting down. 2024-02-06T21:47:13.708 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_2_[41313]: starting. 2024-02-06T21:47:13.708 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_2_[41313]: deleting pool ceph_test_rados_delete_pools_parallel.smithi120-41265 2024-02-06T21:47:13.709 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_2_[41313]: shutting down. 2024-02-06T21:47:13.709 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: ******************************* 2024-02-06T21:47:13.709 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_3_[72609]: starting. 2024-02-06T21:47:13.709 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_3_[72609]: creating pool ceph_test_rados_delete_pools_parallel.smithi120-41265 2024-02-06T21:47:13.709 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_3_[72609]: created object 0... 2024-02-06T21:47:13.709 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_3_[72609]: created object 25... 2024-02-06T21:47:13.709 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_3_[72609]: created object 49... 2024-02-06T21:47:13.709 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_3_[72609]: finishing. 2024-02-06T21:47:13.709 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_3_[72609]: shutting down. 2024-02-06T21:47:13.709 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: ******************************* 2024-02-06T21:47:13.709 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_5_[72614]: starting. 2024-02-06T21:47:13.709 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_5_[72614]: listing objects. 2024-02-06T21:47:13.709 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_5_[72614]: listed object 0... 2024-02-06T21:47:13.709 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_5_[72614]: listed object 25... 2024-02-06T21:47:13.709 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_5_[72614]: saw 50 objects 2024-02-06T21:47:13.710 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_5_[72614]: shutting down. 2024-02-06T21:47:13.710 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: ******************************* 2024-02-06T21:47:13.710 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_4_[72612]: starting. 2024-02-06T21:47:13.710 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_4_[72612]: deleting pool ceph_test_rados_delete_pools_parallel.smithi120-41265 2024-02-06T21:47:13.710 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: process_4_[72612]: shutting down. 2024-02-06T21:47:13.710 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: ******************************* 2024-02-06T21:47:13.710 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: ******************************* 2024-02-06T21:47:13.710 INFO:tasks.workunit.client.0.smithi120.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-02-06T21:47:13.776 INFO:tasks.workunit.client.0.smithi120.stdout:bRadosMisc.Exec (6834 ms) 2024-02-06T21:47:13.776 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-02-06T21:47:13.776 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (87445 ms) 2024-02-06T21:47:13.776 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-02-06T21:47:13.776 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (27447 ms) 2024-02-06T21:47:13.776 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-02-06T21:47:13.777 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5329 ms) 2024-02-06T21:47:13.777 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-02-06T21:47:13.777 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-02-06T21:47:13.777 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-02-06T21:47:13.777 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-02-06T21:47:13.777 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-02-06T21:47:13.777 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (74613 ms) 2024-02-06T21:47:13.777 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [----------] 8 tests from LibRadosMisc (201668 ms total) 2024-02-06T21:47:13.777 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: 2024-02-06T21:47:13.777 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [----------] Global test environment tear-down 2024-02-06T21:47:13.778 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (278159 ms total) 2024-02-06T21:47:13.778 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc: [ PASSED ] 12 tests. 2024-02-06T21:47:13.863 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:55.948873+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi120-41645-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-06T21:47:13.863 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:55.610644+0000 mon.c [INF] from='client.? 172.21.15.120:0/1698989183' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-06T21:47:13.863 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:57.926596+0000 client.admin [INF] threexx 2024-02-06T21:47:13.863 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: got: 2024-02-06T21:46:58.008881+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-06T21:47:13.864 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (47286 ms) 2024-02-06T21:47:13.864 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (278114 ms total) 2024-02-06T21:47:13.864 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: 2024-02-06T21:47:13.864 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: [----------] Global test environment tear-down 2024-02-06T21:47:13.864 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (278114 ms total) 2024-02-06T21:47:13.864 INFO:tasks.workunit.client.0.smithi120.stdout: api_cmd: [ PASSED ] 4 tests. 2024-02-06T21:47:13.897 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: Running main() from gmock_main.cc 2024-02-06T21:47:13.897 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-02-06T21:47:13.897 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [----------] Global test environment set-up. 2024-02-06T21:47:13.897 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-02-06T21:47:13.897 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-02-06T21:47:13.897 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (70466 ms) 2024-02-06T21:47:13.898 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-02-06T21:47:13.898 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-02-06T21:47:13.898 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-02-06T21:47:13.898 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: no timeout :/ 2024-02-06T21:47:13.898 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: no timeout :/ 2024-02-06T21:47:13.898 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: no timeout :/ 2024-02-06T21:47:13.898 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (11661 ms) 2024-02-06T21:47:13.898 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-02-06T21:47:13.898 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (66640 ms) 2024-02-06T21:47:13.898 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-02-06T21:47:13.898 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIo.Checksum (5143 ms) 2024-02-06T21:47:13.898 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-02-06T21:47:13.898 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (10835 ms) 2024-02-06T21:47:13.898 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-02-06T21:47:13.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (13752 ms) 2024-02-06T21:47:13.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-02-06T21:47:13.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (4812 ms) 2024-02-06T21:47:13.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-02-06T21:47:13.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2366 ms) 2024-02-06T21:47:13.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-02-06T21:47:13.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIo.TruncTest (2698 ms) 2024-02-06T21:47:13.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-02-06T21:47:13.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (21341 ms) 2024-02-06T21:47:13.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-02-06T21:47:13.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (1154 ms) 2024-02-06T21:47:13.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-02-06T21:47:13.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIo.RmXattr (233 ms) 2024-02-06T21:47:13.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-02-06T21:47:13.899 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIo.XattrIter (94 ms) 2024-02-06T21:47:13.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [----------] 14 tests from LibRadosIo (211196 ms total) 2024-02-06T21:47:13.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: 2024-02-06T21:47:13.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-02-06T21:47:13.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-02-06T21:47:13.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (16855 ms) 2024-02-06T21:47:13.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-02-06T21:47:13.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (675 ms) 2024-02-06T21:47:13.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-02-06T21:47:13.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (422 ms) 2024-02-06T21:47:13.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-02-06T21:47:13.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (2908 ms) 2024-02-06T21:47:13.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-02-06T21:47:13.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (747 ms) 2024-02-06T21:47:13.900 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-02-06T21:47:13.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (301 ms) 2024-02-06T21:47:13.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-02-06T21:47:13.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (225 ms) 2024-02-06T21:47:13.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-02-06T21:47:13.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (1849 ms) 2024-02-06T21:47:13.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-02-06T21:47:13.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (958 ms) 2024-02-06T21:47:13.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-02-06T21:47:13.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (224 ms) 2024-02-06T21:47:13.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [----------] 10 tests from LibRadosIoEC (25164 ms total) 2024-02-06T21:47:13.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: 2024-02-06T21:47:13.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [----------] Global test environment tear-down 2024-02-06T21:47:13.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [==========] 24 tests from 2 test suites ran. (278372 ms total) 2024-02-06T21:47:13.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_io: [ PASSED ] 24 tests. 2024-02-06T21:47:28.808 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: Running main() from gmock_main.cc 2024-02-06T21:47:28.808 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-02-06T21:47:28.809 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [----------] Global test environment set-up. 2024-02-06T21:47:28.809 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-02-06T21:47:28.809 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: seed 40537 2024-02-06T21:47:28.809 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-02-06T21:47:28.809 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-02-06T21:47:28.809 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-02-06T21:47:28.809 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (121022 ms) 2024-02-06T21:47:28.809 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-02-06T21:47:28.809 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (1681 ms) 2024-02-06T21:47:28.809 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-02-06T21:47:28.809 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (2773 ms) 2024-02-06T21:47:28.809 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-02-06T21:47:28.809 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (47330 ms) 2024-02-06T21:47:28.810 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-02-06T21:47:28.810 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (2606 ms) 2024-02-06T21:47:28.810 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-02-06T21:47:28.810 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (3575 ms) 2024-02-06T21:47:28.810 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-02-06T21:47:28.810 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (2342 ms) 2024-02-06T21:47:28.810 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-02-06T21:47:28.810 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (6970 ms) 2024-02-06T21:47:28.810 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-02-06T21:47:28.811 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (22501 ms) 2024-02-06T21:47:28.811 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-02-06T21:47:28.811 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (75 ms) 2024-02-06T21:47:28.811 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-02-06T21:47:28.811 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (4326 ms) 2024-02-06T21:47:28.811 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-02-06T21:47:28.811 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (96 ms) 2024-02-06T21:47:28.811 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-02-06T21:47:28.811 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (95 ms) 2024-02-06T21:47:28.811 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-02-06T21:47:28.811 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (81 ms) 2024-02-06T21:47:28.811 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-02-06T21:47:28.811 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (548 ms) 2024-02-06T21:47:28.811 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-02-06T21:47:28.811 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (364 ms) 2024-02-06T21:47:28.812 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-02-06T21:47:28.812 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (167 ms) 2024-02-06T21:47:28.812 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-02-06T21:47:28.812 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (303 ms) 2024-02-06T21:47:28.812 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-02-06T21:47:28.812 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (69 ms) 2024-02-06T21:47:28.812 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (216925 ms total) 2024-02-06T21:47:28.812 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: 2024-02-06T21:47:28.812 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-02-06T21:47:28.812 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-02-06T21:47:28.812 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (6338 ms) 2024-02-06T21:47:28.812 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-02-06T21:47:28.812 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (7013 ms) 2024-02-06T21:47:28.813 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-02-06T21:47:28.813 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (3605 ms) 2024-02-06T21:47:28.813 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-02-06T21:47:28.813 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (3884 ms) 2024-02-06T21:47:28.813 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-02-06T21:47:29.196 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:07:09.010 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:29.255 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:07:09.070 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:29.340 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:07:09.154 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:29.388 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:07:09.202 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:29.394 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:07:09.209 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:29.426 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:09.190 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:29.437 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:09.201 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:29.488 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:07:09.302 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:29.522 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:09.286 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:49.334 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:07:29.148 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:49.339 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:07:29.154 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:49.407 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:07:29.222 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:49.408 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:29.171 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:49.411 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:29.175 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:49.414 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:07:29.229 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:49.446 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:07:29.261 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:49.505 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:07:29.319 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:47:49.506 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:29.271 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:07.585 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.Round api_snapshots_pp: Running main() from gmock_main.cc 2024-02-06T21:48:07.586 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-02-06T21:48:07.586 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-02-06T21:48:07.586 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-02-06T21:48:07.586 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: seed 40778 2024-02-06T21:48:07.586 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-02-06T21:48:07.586 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (27282 ms) 2024-02-06T21:48:07.586 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-02-06T21:48:07.586 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (11447 ms) 2024-02-06T21:48:07.586 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-02-06T21:48:07.586 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (45582 ms) 2024-02-06T21:48:07.586 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-02-06T21:48:07.586 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (45493 ms) 2024-02-06T21:48:07.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-02-06T21:48:07.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (10068 ms) 2024-02-06T21:48:07.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (139873 ms total) 2024-02-06T21:48:07.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: 2024-02-06T21:48:07.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-02-06T21:48:07.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-02-06T21:48:07.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (28032 ms) 2024-02-06T21:48:07.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-02-06T21:48:07.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (34753 ms) 2024-02-06T21:48:07.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-02-06T21:48:07.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (60433 ms) 2024-02-06T21:48:07.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-02-06T21:48:07.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (12716 ms) 2024-02-06T21:48:07.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-02-06T21:48:07.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (6503 ms) 2024-02-06T21:48:07.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-02-06T21:48:07.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1269-g633ab857/rpm/el9/BUILD/ceph-19.0.0-1269-g633ab857/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-02-06T21:48:07.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: 2024-02-06T21:48:07.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-02-06T21:48:07.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-02-06T21:48:07.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi120-40778-7 2024-02-06T21:48:07.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: waiting for snaps to purge 2024-02-06T21:48:07.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (20185 ms) 2024-02-06T21:48:07.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (162622 ms total) 2024-02-06T21:48:07.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: 2024-02-06T21:48:07.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-02-06T21:48:07.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-02-06T21:48:07.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (7 ms) 2024-02-06T21:48:07.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-02-06T21:48:07.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (6112 ms) 2024-02-06T21:48:07.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (6119 ms total) 2024-02-06T21:48:07.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: 2024-02-06T21:48:07.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-02-06T21:48:07.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-02-06T21:48:07.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (7134 ms) 2024-02-06T21:48:07.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-02-06T21:48:08.813 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:07:48.628 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:08.819 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:07:48.634 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:08.897 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:48.661 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:08.903 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:07:48.718 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:08.906 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:07:48.721 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:08.907 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:48.671 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:08.920 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:07:48.735 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:09.011 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:48.775 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:09.021 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:07:48.836 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:10.266 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:50.030 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:10.267 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:50.032 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:10.283 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:50.047 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:10.596 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:07:50.410 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:10.602 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:07:50.417 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:10.624 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:07:50.439 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:10.771 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:07:50.585 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:10.772 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:07:50.587 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:10.794 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:07:50.608 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:15.073 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK io: Running main() from gmock_main.cc 2024-02-06T21:48:15.073 INFO:tasks.workunit.client.0.smithi120.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-02-06T21:48:15.073 INFO:tasks.workunit.client.0.smithi120.stdout: io: [----------] Global test environment set-up. 2024-02-06T21:48:15.073 INFO:tasks.workunit.client.0.smithi120.stdout: io: [----------] 14 tests from NeoRadosIo 2024-02-06T21:48:15.073 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-02-06T21:48:15.073 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ OK ] NeoRadosIo.Limits (28667 ms) 2024-02-06T21:48:15.073 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-02-06T21:48:15.073 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (34145 ms) 2024-02-06T21:48:15.074 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-02-06T21:48:15.074 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ OK ] NeoRadosIo.ReadOp (10733 ms) 2024-02-06T21:48:15.074 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-02-06T21:48:15.074 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ OK ] NeoRadosIo.SparseRead (93867 ms) 2024-02-06T21:48:15.074 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-02-06T21:48:15.074 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ OK ] NeoRadosIo.RoundTrip (37476 ms) 2024-02-06T21:48:15.074 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-02-06T21:48:15.074 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (32215 ms) 2024-02-06T21:48:15.074 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-02-06T21:48:15.074 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (35018 ms) 2024-02-06T21:48:15.074 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-02-06T21:48:15.074 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (15319 ms) 2024-02-06T21:48:15.074 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-02-06T21:48:15.075 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (5737 ms) 2024-02-06T21:48:15.075 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-02-06T21:48:15.075 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ OK ] NeoRadosIo.Trunc (10204 ms) 2024-02-06T21:48:15.075 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-02-06T21:48:15.075 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ OK ] NeoRadosIo.Remove (15514 ms) 2024-02-06T21:48:15.075 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-02-06T21:48:15.075 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (6072 ms) 2024-02-06T21:48:15.075 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-02-06T21:48:15.075 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ OK ] NeoRadosIo.RmXattr (4444 ms) 2024-02-06T21:48:15.075 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-02-06T21:48:15.075 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ OK ] NeoRadosIo.GetXattrs (9688 ms) 2024-02-06T21:48:15.075 INFO:tasks.workunit.client.0.smithi120.stdout: io: [----------] 14 tests from NeoRadosIo (339099 ms total) 2024-02-06T21:48:15.075 INFO:tasks.workunit.client.0.smithi120.stdout: io: 2024-02-06T21:48:15.076 INFO:tasks.workunit.client.0.smithi120.stdout: io: [----------] Global test environment tear-down 2024-02-06T21:48:15.076 INFO:tasks.workunit.client.0.smithi120.stdout: io: [==========] 14 tests from 1 test suite ran. (339099 ms total) 2024-02-06T21:48:15.076 INFO:tasks.workunit.client.0.smithi120.stdout: io: [ PASSED ] 14 tests. 2024-02-06T21:48:16.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: Running main() from gmock_main.cc 2024-02-06T21:48:16.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-02-06T21:48:16.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [----------] Global test environment set-up. 2024-02-06T21:48:16.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-02-06T21:48:16.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-02-06T21:48:16.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (98811 ms) 2024-02-06T21:48:16.230 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-02-06T21:48:16.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (20968 ms) 2024-02-06T21:48:16.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-02-06T21:48:16.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (22864 ms) 2024-02-06T21:48:16.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-02-06T21:48:16.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (8182 ms) 2024-02-06T21:48:16.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (150825 ms total) 2024-02-06T21:48:16.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: 2024-02-06T21:48:16.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-02-06T21:48:16.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-02-06T21:48:16.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (22987 ms) 2024-02-06T21:48:16.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-02-06T21:48:16.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (45499 ms) 2024-02-06T21:48:16.231 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-02-06T21:48:16.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (52130 ms) 2024-02-06T21:48:16.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (120616 ms total) 2024-02-06T21:48:16.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: 2024-02-06T21:48:16.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-02-06T21:48:16.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-02-06T21:48:16.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (6504 ms) 2024-02-06T21:48:16.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-02-06T21:48:16.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (4907 ms) 2024-02-06T21:48:16.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-02-06T21:48:16.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (8030 ms) 2024-02-06T21:48:16.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-02-06T21:48:16.232 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (4138 ms) 2024-02-06T21:48:16.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (23579 ms total) 2024-02-06T21:48:16.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: 2024-02-06T21:48:16.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-02-06T21:48:16.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-02-06T21:48:16.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (13577 ms) 2024-02-06T21:48:16.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-02-06T21:48:16.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (10904 ms) 2024-02-06T21:48:16.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (24481 ms total) 2024-02-06T21:48:16.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: 2024-02-06T21:48:16.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [----------] Global test environment tear-down 2024-02-06T21:48:16.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (340623 ms total) 2024-02-06T21:48:16.233 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-02-06T21:48:18.204 INFO:tasks.workunit.client.0.smithi120.stdout:********* 2024-02-06T21:48:18.204 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_9_[86457]: starting. 2024-02-06T21:48:18.205 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_9_[86457]: creating pool ceph_test_rados_list_parallel.smithi120-41274 2024-02-06T21:48:18.205 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_9_[86457]: created object 0... 2024-02-06T21:48:18.205 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_9_[86457]: created object 25... 2024-02-06T21:48:18.205 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_9_[86457]: created object 49... 2024-02-06T21:48:18.205 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_9_[86457]: finishing. 2024-02-06T21:48:18.205 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_9_[86457]: shutting down. 2024-02-06T21:48:18.205 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.205 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.205 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.205 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_10_[86458]: starting. 2024-02-06T21:48:18.205 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_10_[86458]: listing objects. 2024-02-06T21:48:18.205 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_10_[86458]: listed object 0... 2024-02-06T21:48:18.205 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_10_[86458]: listed object 25... 2024-02-06T21:48:18.205 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_10_[86458]: listed object 50... 2024-02-06T21:48:18.205 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_10_[86458]: listed object 75... 2024-02-06T21:48:18.206 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_10_[86458]: saw 98 objects 2024-02-06T21:48:18.206 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_10_[86458]: shutting down. 2024-02-06T21:48:18.206 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.206 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.206 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.206 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_13_[86461]: starting. 2024-02-06T21:48:18.206 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_13_[86461]: removed 25 objects... 2024-02-06T21:48:18.206 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_13_[86461]: removed half of the objects 2024-02-06T21:48:18.206 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_13_[86461]: removed 50 objects... 2024-02-06T21:48:18.206 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_13_[86461]: removed 50 objects 2024-02-06T21:48:18.206 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_13_[86461]: shutting down. 2024-02-06T21:48:18.206 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.206 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.206 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.206 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_12_[86460]: starting. 2024-02-06T21:48:18.207 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_12_[86460]: added 25 objects... 2024-02-06T21:48:18.207 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_12_[86460]: added half of the objects 2024-02-06T21:48:18.207 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_12_[86460]: added 50 objects... 2024-02-06T21:48:18.207 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_12_[86460]: added 50 objects 2024-02-06T21:48:18.207 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_12_[86460]: shutting down. 2024-02-06T21:48:18.207 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.207 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.207 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.207 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_11_[86459]: starting. 2024-02-06T21:48:18.207 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_11_[86459]: added 25 objects... 2024-02-06T21:48:18.207 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_11_[86459]: added half of the objects 2024-02-06T21:48:18.207 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_11_[86459]: added 50 objects... 2024-02-06T21:48:18.207 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_11_[86459]: added 50 objects 2024-02-06T21:48:18.207 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_11_[86459]: shutting down. 2024-02-06T21:48:18.207 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.208 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.208 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.208 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.208 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_14_[100482]: starting. 2024-02-06T21:48:18.208 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_14_[100482]: creating pool ceph_test_rados_list_parallel.smithi120-41274 2024-02-06T21:48:18.208 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_14_[100482]: created object 0... 2024-02-06T21:48:18.208 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_14_[100482]: created object 25... 2024-02-06T21:48:18.208 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_14_[100482]: created object 49... 2024-02-06T21:48:18.208 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_14_[100482]: finishing. 2024-02-06T21:48:18.208 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_14_[100482]: shutting down. 2024-02-06T21:48:18.208 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.208 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.208 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.208 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:18.208 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_15_[100484]: starting. 2024-02-06T21:48:18.209 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_15_[100484]: listing objects. 2024-02-06T21:48:18.209 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_15_[100484]: listed object 0... 2024-02-06T21:48:18.433 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_15_[100484]: list snapshots: Running main() from gmock_main.cc 2024-02-06T21:48:18.433 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-02-06T21:48:18.433 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [----------] Global test environment set-up. 2024-02-06T21:48:18.433 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-02-06T21:48:18.433 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-02-06T21:48:18.433 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (30005 ms) 2024-02-06T21:48:18.434 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-02-06T21:48:18.434 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (27647 ms) 2024-02-06T21:48:18.434 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-02-06T21:48:18.434 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (24456 ms) 2024-02-06T21:48:18.434 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-02-06T21:48:18.434 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (82426 ms) 2024-02-06T21:48:18.434 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-02-06T21:48:18.434 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (20404 ms) 2024-02-06T21:48:18.434 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (184938 ms total) 2024-02-06T21:48:18.434 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: 2024-02-06T21:48:18.434 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-02-06T21:48:18.435 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-02-06T21:48:18.435 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (15136 ms) 2024-02-06T21:48:18.435 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-02-06T21:48:18.435 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (41577 ms) 2024-02-06T21:48:18.435 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-02-06T21:48:18.435 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (54196 ms) 2024-02-06T21:48:18.435 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-02-06T21:48:18.435 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (8517 ms) 2024-02-06T21:48:18.435 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-02-06T21:48:18.435 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (11164 ms) 2024-02-06T21:48:18.436 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-02-06T21:48:18.436 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi120-41698-11. 2024-02-06T21:48:18.436 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: Waiting for snaps to purge. 2024-02-06T21:48:18.436 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (26864 ms) 2024-02-06T21:48:18.436 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (157454 ms total) 2024-02-06T21:48:18.436 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: 2024-02-06T21:48:18.436 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [----------] Global test environment tear-down 2024-02-06T21:48:18.436 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (342392 ms total) 2024-02-06T21:48:18.436 INFO:tasks.workunit.client.0.smithi120.stdout: snapshots: [ PASSED ] 11 tests. 2024-02-06T21:48:24.283 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: Running main() from gmock_main.cc 2024-02-06T21:48:24.283 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-02-06T21:48:24.283 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [----------] Global test environment set-up. 2024-02-06T21:48:24.283 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-02-06T21:48:24.283 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-02-06T21:48:24.283 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (100427 ms) 2024-02-06T21:48:24.284 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-02-06T21:48:24.284 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (69226 ms) 2024-02-06T21:48:24.284 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-02-06T21:48:24.284 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (10713 ms) 2024-02-06T21:48:24.284 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-02-06T21:48:24.284 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (38020 ms) 2024-02-06T21:48:24.284 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-02-06T21:48:24.284 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (36775 ms) 2024-02-06T21:48:24.284 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-02-06T21:48:24.284 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (22507 ms) 2024-02-06T21:48:24.284 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-02-06T21:48:24.284 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (10786 ms) 2024-02-06T21:48:24.284 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-02-06T21:48:24.285 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (4656 ms) 2024-02-06T21:48:24.285 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-02-06T21:48:24.285 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (9359 ms) 2024-02-06T21:48:24.285 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-02-06T21:48:24.285 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (6983 ms) 2024-02-06T21:48:24.285 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-02-06T21:48:24.285 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (12928 ms) 2024-02-06T21:48:24.285 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-02-06T21:48:24.285 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (13901 ms) 2024-02-06T21:48:24.285 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-02-06T21:48:24.285 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (6110 ms) 2024-02-06T21:48:24.285 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-02-06T21:48:24.285 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (5867 ms) 2024-02-06T21:48:24.285 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (348258 ms total) 2024-02-06T21:48:24.285 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: 2024-02-06T21:48:24.286 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [----------] Global test environment tear-down 2024-02-06T21:48:24.286 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (348258 ms total) 2024-02-06T21:48:24.286 INFO:tasks.workunit.client.0.smithi120.stdout: read_operations: [ PASSED ] 14 tests. 2024-02-06T21:48:30.202 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:08:10.017 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:30.207 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:08:10.021 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:30.211 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:08:10.025 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:30.214 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:08:10.029 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:30.218 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:08:09.982 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:30.221 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:08:09.985 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:30.365 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:08:10.179 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:30.365 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:08:10.179 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:30.391 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:08:10.155 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:31.516 INFO:tasks.workunit.client.0.smithi120.stdout:TripPP2 (2075 ms) 2024-02-06T21:48:31.516 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-02-06T21:48:31.516 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (636 ms) 2024-02-06T21:48:31.516 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-02-06T21:48:31.516 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (9701 ms) 2024-02-06T21:48:31.516 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-02-06T21:48:31.516 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (3871 ms) 2024-02-06T21:48:31.516 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-02-06T21:48:31.516 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (5659 ms) 2024-02-06T21:48:31.516 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-02-06T21:48:31.517 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (4080 ms) 2024-02-06T21:48:31.517 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-02-06T21:48:31.517 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3103 ms) 2024-02-06T21:48:31.517 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-02-06T21:48:31.517 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (6608 ms) 2024-02-06T21:48:31.517 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-02-06T21:48:31.517 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (15607 ms) 2024-02-06T21:48:31.517 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-02-06T21:48:31.517 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (2911 ms) 2024-02-06T21:48:31.517 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-02-06T21:48:31.517 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (2550 ms) 2024-02-06T21:48:31.517 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-02-06T21:48:31.517 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (1270 ms) 2024-02-06T21:48:31.517 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-02-06T21:48:31.518 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (3017 ms) 2024-02-06T21:48:31.518 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (81929 ms total) 2024-02-06T21:48:31.518 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: 2024-02-06T21:48:31.518 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [----------] Global test environment tear-down 2024-02-06T21:48:31.518 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (355971 ms total) 2024-02-06T21:48:31.518 INFO:tasks.workunit.client.0.smithi120.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-02-06T21:48:34.484 INFO:tasks.workunit.client.0.smithi120.stdout:ed object 25... 2024-02-06T21:48:34.484 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_15_[100484]: listed object 50... 2024-02-06T21:48:34.484 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_15_[100484]: listed object 75... 2024-02-06T21:48:34.485 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_15_[100484]: listed object 100... 2024-02-06T21:48:34.485 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_15_[100484]: listed object 125... 2024-02-06T21:48:34.485 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_15_[100484]: saw 150 objects 2024-02-06T21:48:34.485 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_15_[100484]: shutting down. 2024-02-06T21:48:34.485 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:34.485 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:34.485 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:34.485 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:34.485 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_16_[100485]: starting. 2024-02-06T21:48:34.485 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_16_[100485]: added 25 objects... 2024-02-06T21:48:34.485 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_16_[100485]: added half of the objects 2024-02-06T21:48:34.485 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_16_[100485]: added 50 objects... 2024-02-06T21:48:34.485 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_16_[100485]: added 50 objects 2024-02-06T21:48:34.486 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: process_16_[100485]: shutting down. 2024-02-06T21:48:34.486 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:34.486 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:34.486 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:34.486 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:34.486 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******************************* 2024-02-06T21:48:34.486 INFO:tasks.workunit.client.0.smithi120.stdout: list_parallel: ******* SUCCESS ********** 2024-02-06T21:48:34.486 INFO:tasks.workunit.client.0.smithi120.stdout: misc: Running main() from gmock_main.cc 2024-02-06T21:48:34.486 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-02-06T21:48:34.487 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [----------] Global test environment set-up. 2024-02-06T21:48:34.487 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-02-06T21:48:34.487 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-02-06T21:48:34.487 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ OK ] NeoRadosMisc.Version (4511 ms) 2024-02-06T21:48:34.487 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-02-06T21:48:34.487 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (4176 ms) 2024-02-06T21:48:34.487 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-02-06T21:48:34.487 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ OK ] NeoRadosMisc.LongName (251600 ms) 2024-02-06T21:48:34.487 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-02-06T21:48:34.487 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (26029 ms) 2024-02-06T21:48:34.487 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-02-06T21:48:34.487 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (8151 ms) 2024-02-06T21:48:34.487 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-02-06T21:48:34.487 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (17134 ms) 2024-02-06T21:48:34.488 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-02-06T21:48:34.488 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ OK ] NeoRadosMisc.Exec (7278 ms) 2024-02-06T21:48:34.488 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-02-06T21:48:34.488 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (20188 ms) 2024-02-06T21:48:34.488 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-02-06T21:48:34.488 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (8165 ms) 2024-02-06T21:48:34.488 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-02-06T21:48:34.488 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ OK ] NeoRadosMisc.BigObject (4503 ms) 2024-02-06T21:48:34.488 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-02-06T21:48:34.488 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2137 ms) 2024-02-06T21:48:34.488 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-02-06T21:48:34.488 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (4608 ms) 2024-02-06T21:48:34.488 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [----------] 12 tests from NeoRadosMisc (358482 ms total) 2024-02-06T21:48:34.488 INFO:tasks.workunit.client.0.smithi120.stdout: misc: 2024-02-06T21:48:34.488 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [----------] Global test environment tear-down 2024-02-06T21:48:34.489 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [==========] 12 tests from 1 test suite ran. (358482 ms total) 2024-02-06T21:48:34.489 INFO:tasks.workunit.client.0.smithi120.stdout: misc: [ PASSED ] 12 tests. 2024-02-06T21:48:48.414 INFO:tasks.workunit.client.0.smithi120.stdout:] LibRadosSnapshotsECPP.SnapRemovePP (3249 ms) 2024-02-06T21:48:48.414 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-02-06T21:48:48.414 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (5732 ms) 2024-02-06T21:48:48.414 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-02-06T21:48:48.414 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (4069 ms) 2024-02-06T21:48:48.414 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (20184 ms total) 2024-02-06T21:48:48.414 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: 2024-02-06T21:48:48.415 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-02-06T21:48:48.415 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-02-06T21:48:48.415 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (7184 ms) 2024-02-06T21:48:48.415 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-02-06T21:48:48.415 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (8899 ms) 2024-02-06T21:48:48.415 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-02-06T21:48:48.415 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (6044 ms) 2024-02-06T21:48:48.415 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (22127 ms total) 2024-02-06T21:48:48.415 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: 2024-02-06T21:48:48.415 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-02-06T21:48:48.415 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (372767 ms total) 2024-02-06T21:48:48.415 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-02-06T21:48:48.415 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-02-06T21:48:48.415 INFO:tasks.workunit.client.0.smithi120.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-02-06T21:48:50.361 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:08:30.176 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:50.368 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:08:30.183 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:50.426 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:08:30.241 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:50.430 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:08:30.245 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:50.434 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:08:30.198 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:50.440 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:08:30.204 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:50.474 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:08:30.289 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:50.497 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:08:30.312 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:48:50.528 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:08:30.292 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:05.396 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: Running main() from gmock_main.cc 2024-02-06T21:49:05.396 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-02-06T21:49:05.396 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [----------] Global test environment set-up. 2024-02-06T21:49:05.396 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-02-06T21:49:05.396 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-02-06T21:49:05.396 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.TooBig (6461 ms) 2024-02-06T21:49:05.396 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-02-06T21:49:05.396 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (126116 ms) 2024-02-06T21:49:05.396 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (7165 ms) 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (30377 ms) 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (10787 ms) 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (45809 ms) 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (37880 ms) 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (21187 ms) 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (7846 ms) 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (13478 ms) 2024-02-06T21:49:05.397 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-02-06T21:49:05.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (6266 ms) 2024-02-06T21:49:05.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-02-06T21:49:05.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (5990 ms) 2024-02-06T21:49:05.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-02-06T21:49:05.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (6061 ms) 2024-02-06T21:49:05.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-02-06T21:49:05.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (10243 ms) 2024-02-06T21:49:05.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-02-06T21:49:05.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.Flush (7208 ms) 2024-02-06T21:49:05.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-02-06T21:49:05.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (5911 ms) 2024-02-06T21:49:05.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-02-06T21:49:05.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3435 ms) 2024-02-06T21:49:05.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-02-06T21:49:05.398 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (5716 ms) 2024-02-06T21:49:05.399 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-02-06T21:49:05.399 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (4833 ms) 2024-02-06T21:49:05.399 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-02-06T21:49:05.399 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (4204 ms) 2024-02-06T21:49:05.399 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-02-06T21:49:05.399 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3799 ms) 2024-02-06T21:49:05.399 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-02-06T21:49:05.399 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3972 ms) 2024-02-06T21:49:05.399 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-02-06T21:49:05.399 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3947 ms) 2024-02-06T21:49:05.399 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-02-06T21:49:05.399 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3553 ms) 2024-02-06T21:49:05.399 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-02-06T21:49:05.399 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (4272 ms) 2024-02-06T21:49:05.399 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-02-06T21:49:05.400 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3364 ms) 2024-02-06T21:49:05.400 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [----------] 26 tests from LibRadosAio (389881 ms total) 2024-02-06T21:49:05.400 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: 2024-02-06T21:49:05.400 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-02-06T21:49:10.469 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:08:50.284 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:10.478 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:08:50.293 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:10.542 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:08:50.306 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:10.545 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:08:50.309 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:10.555 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:08:50.370 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:10.568 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:08:50.382 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:10.599 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:08:50.413 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:10.602 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:08:50.366 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:10.642 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:08:50.457 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:30.642 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:09:10.457 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:30.710 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:09:10.525 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:30.714 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:09:10.478 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:30.718 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:09:10.482 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:30.795 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:09:10.610 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:30.798 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:09:10.612 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:30.801 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:09:10.616 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:30.825 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:09:10.589 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:30.877 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:09:10.691 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:35.031 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.Sim list: Running main() from gmock_main.cc 2024-02-06T21:49:35.031 INFO:tasks.workunit.client.0.smithi120.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-02-06T21:49:35.031 INFO:tasks.workunit.client.0.smithi120.stdout: list: [----------] Global test environment set-up. 2024-02-06T21:49:35.032 INFO:tasks.workunit.client.0.smithi120.stdout: list: [----------] 3 tests from NeoradosList 2024-02-06T21:49:35.032 INFO:tasks.workunit.client.0.smithi120.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-02-06T21:49:35.032 INFO:tasks.workunit.client.0.smithi120.stdout: list: [ OK ] NeoradosList.ListObjects (137216 ms) 2024-02-06T21:49:35.032 INFO:tasks.workunit.client.0.smithi120.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-02-06T21:49:35.032 INFO:tasks.workunit.client.0.smithi120.stdout: list: [ OK ] NeoradosList.ListObjectsNS (96521 ms) 2024-02-06T21:49:35.032 INFO:tasks.workunit.client.0.smithi120.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-02-06T21:49:35.032 INFO:tasks.workunit.client.0.smithi120.stdout: list: [ OK ] NeoradosList.ListObjectsMany (185303 ms) 2024-02-06T21:49:35.032 INFO:tasks.workunit.client.0.smithi120.stdout: list: [----------] 3 tests from NeoradosList (419040 ms total) 2024-02-06T21:49:35.032 INFO:tasks.workunit.client.0.smithi120.stdout: list: 2024-02-06T21:49:35.032 INFO:tasks.workunit.client.0.smithi120.stdout: list: [----------] Global test environment tear-down 2024-02-06T21:49:35.032 INFO:tasks.workunit.client.0.smithi120.stdout: list: [==========] 3 tests from 1 test suite ran. (419040 ms total) 2024-02-06T21:49:35.032 INFO:tasks.workunit.client.0.smithi120.stdout: list: [ PASSED ] 3 tests. 2024-02-06T21:49:41.758 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-02-06T21:49:41.759 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-02-06T21:49:41.759 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-02-06T21:49:41.759 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-02-06T21:49:41.759 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-02-06T21:49:41.759 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-02-06T21:49:41.759 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-02-06T21:49:41.759 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: 2024-02-06T21:49:41.759 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-02-06T21:49:41.759 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-02-06T21:49:41.759 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (25379 ms) 2024-02-06T21:49:41.759 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-02-06T21:49:41.759 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (52046 ms) 2024-02-06T21:49:41.759 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-02-06T21:49:41.759 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (13265 ms) 2024-02-06T21:49:41.760 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-02-06T21:49:41.760 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (11579 ms) 2024-02-06T21:49:41.760 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-02-06T21:49:41.760 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (13292 ms) 2024-02-06T21:49:41.760 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-02-06T21:49:41.760 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (6641 ms) 2024-02-06T21:49:41.760 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-02-06T21:49:41.760 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (303751 ms) 2024-02-06T21:49:41.760 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (425953 ms total) 2024-02-06T21:49:41.760 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: 2024-02-06T21:49:41.760 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-02-06T21:49:41.760 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (425953 ms total) 2024-02-06T21:49:41.760 INFO:tasks.workunit.client.0.smithi120.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-02-06T21:49:50.711 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:09:30.525 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:50.781 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:09:30.596 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:50.805 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:09:30.569 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:50.819 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:09:30.583 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:50.841 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:09:30.656 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:50.852 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:09:30.667 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:50.878 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:09:30.693 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:50.910 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:09:30.675 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:50.937 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:09:30.751 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:52.960 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:09:32.774 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:53.015 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:09:32.830 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:53.040 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:09:32.855 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:53.080 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:09:32.844 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:53.081 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:09:32.896 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:53.082 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:09:32.846 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:53.083 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:09:32.898 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:53.106 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:09:32.870 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:53.106 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:09:32.921 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:49:53.459 INFO:tasks.workunit.client.0.smithi120.stdout:_list: : entry=3 expected=3 2024-02-06T21:49:53.459 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2024-02-06T21:49:53.459 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:c4fdafeb:::6:head expected=13:c4fdafeb:::6:head 2024-02-06T21:49:53.459 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2024-02-06T21:49:53.459 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : entry=6 expected=6 2024-02-06T21:49:53.459 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2024-02-06T21:49:53.459 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:bd63b0f1:::8:head expected=13:bd63b0f1:::8:head 2024-02-06T21:49:53.459 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2024-02-06T21:49:53.459 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : entry=8 expected=8 2024-02-06T21:49:53.460 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:b29083e3:::5:head 2024-02-06T21:49:53.460 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:b29083e3:::5:head expected=13:b29083e3:::5:head 2024-02-06T21:49:53.460 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:b29083e3:::5:head -> 5 2024-02-06T21:49:53.460 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : entry=5 expected=5 2024-02-06T21:49:53.460 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:89d3ae78:::11:head 2024-02-06T21:49:53.460 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:89d3ae78:::11:head expected=13:89d3ae78:::11:head 2024-02-06T21:49:53.460 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2024-02-06T21:49:53.460 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : entry=11 expected=11 2024-02-06T21:49:53.460 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:863748b0:::15:head 2024-02-06T21:49:53.460 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:863748b0:::15:head expected=13:863748b0:::15:head 2024-02-06T21:49:53.460 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:863748b0:::15:head -> 15 2024-02-06T21:49:53.460 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : entry=15 expected=15 2024-02-06T21:49:53.460 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:6cac518f:::0:head 2024-02-06T21:49:53.461 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:6cac518f:::0:head expected=13:6cac518f:::0:head 2024-02-06T21:49:53.461 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:6cac518f:::0:head -> 0 2024-02-06T21:49:53.461 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : entry=0 expected=0 2024-02-06T21:49:53.461 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:62a1935d:::14:head 2024-02-06T21:49:53.461 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:62a1935d:::14:head expected=13:62a1935d:::14:head 2024-02-06T21:49:53.461 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:62a1935d:::14:head -> 14 2024-02-06T21:49:53.461 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : entry=14 expected=14 2024-02-06T21:49:53.461 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : seek to 13:02547ec2:::1:head 2024-02-06T21:49:53.461 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : cursor()=13:02547ec2:::1:head expected=13:02547ec2:::1:head 2024-02-06T21:49:53.461 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: > 13:02547ec2:::1:head -> 1 2024-02-06T21:49:53.461 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: : entry=1 expected=1 2024-02-06T21:49:53.461 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (11082 ms) 2024-02-06T21:49:53.461 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-02-06T21:49:53.461 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (155737 ms) 2024-02-06T21:49:53.461 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-02-06T21:49:53.462 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: split 0/5 -> MIN 13:33333333::::head 2024-02-06T21:49:53.462 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: split 1/5 -> 13:33333333::::head 13:66666666::::head 2024-02-06T21:49:53.462 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: split 2/5 -> 13:66666666::::head 13:99999999::::head 2024-02-06T21:49:53.462 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: split 3/5 -> 13:99999999::::head 13:cccccccc::::head 2024-02-06T21:49:53.462 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: split 4/5 -> 13:cccccccc::::head MAX 2024-02-06T21:49:53.462 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (21458 ms) 2024-02-06T21:49:53.462 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [----------] 7 tests from LibRadosList (421420 ms total) 2024-02-06T21:49:53.462 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 2024-02-06T21:49:53.462 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-02-06T21:49:53.462 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-02-06T21:49:53.462 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (2590 ms) 2024-02-06T21:49:53.462 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-02-06T21:49:53.462 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: myset foo1,foo2,foo3 2024-02-06T21:49:53.462 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo1 2024-02-06T21:49:53.463 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo2 2024-02-06T21:49:53.463 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo3 2024-02-06T21:49:53.463 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: myset foo1,foo4,foo5 2024-02-06T21:49:53.463 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo4 2024-02-06T21:49:53.463 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo5 2024-02-06T21:49:53.463 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo1 2024-02-06T21:49:53.463 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: myset foo6,foo7 2024-02-06T21:49:53.463 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo7 2024-02-06T21:49:53.463 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: foo6 2024-02-06T21:49:53.463 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-02-06T21:49:53.463 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: ns1:foo4 2024-02-06T21:49:53.463 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: ns1:foo5 2024-02-06T21:49:53.463 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: ns2:foo7 2024-02-06T21:49:53.463 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: ns2:foo6 2024-02-06T21:49:53.464 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: ns1:foo1 2024-02-06T21:49:53.464 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: :foo1 2024-02-06T21:49:53.464 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: :foo2 2024-02-06T21:49:53.464 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: :foo3 2024-02-06T21:49:53.464 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (3764 ms) 2024-02-06T21:50:03.141 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-02-06T21:50:03.142 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 1 0 2024-02-06T21:50:03.142 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 10 0 2024-02-06T21:50:03.142 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 13 0 2024-02-06T21:50:03.142 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 7 0 2024-02-06T21:50:03.142 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 14 0 2024-02-06T21:50:03.142 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 0 0 2024-02-06T21:50:03.142 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 15 0 2024-02-06T21:50:03.142 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 11 0 2024-02-06T21:50:03.142 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 5 0 2024-02-06T21:50:03.142 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 8 0 2024-02-06T21:50:03.142 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 6 0 2024-02-06T21:50:03.143 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 3 0 2024-02-06T21:50:03.143 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 4 0 2024-02-06T21:50:03.143 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 12 0 2024-02-06T21:50:03.143 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 9 0 2024-02-06T21:50:03.143 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 2 0 2024-02-06T21:50:03.143 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-02-06T21:50:03.143 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (3801 ms) 2024-02-06T21:50:03.143 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [----------] 3 tests from LibRadosListEC (10155 ms total) 2024-02-06T21:50:03.143 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 2024-02-06T21:50:03.143 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-02-06T21:50:03.143 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-02-06T21:50:03.143 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2763 ms) 2024-02-06T21:50:03.143 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [----------] 1 test from LibRadosListNP (2763 ms total) 2024-02-06T21:50:03.143 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: 2024-02-06T21:50:03.144 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [----------] Global test environment tear-down 2024-02-06T21:50:03.144 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [==========] 11 tests from 3 test suites ran. (447579 ms total) 2024-02-06T21:50:03.144 INFO:tasks.workunit.client.0.smithi120.stdout: api_list: [ PASSED ] 11 tests. 2024-02-06T21:50:05.372 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: Running main() from gmock_main.cc 2024-02-06T21:50:05.372 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-02-06T21:50:05.372 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [----------] Global test environment set-up. 2024-02-06T21:50:05.372 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-02-06T21:50:05.372 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-02-06T21:50:05.372 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (78682 ms) 2024-02-06T21:50:05.372 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-02-06T21:50:05.372 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (17767 ms) 2024-02-06T21:50:05.372 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-02-06T21:50:05.372 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (42780 ms) 2024-02-06T21:50:05.372 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-02-06T21:50:05.373 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (25304 ms) 2024-02-06T21:50:05.373 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-02-06T21:50:05.373 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (20413 ms) 2024-02-06T21:50:05.373 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-02-06T21:50:05.373 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (41205 ms) 2024-02-06T21:50:05.373 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-02-06T21:50:05.373 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (223181 ms) 2024-02-06T21:50:05.373 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (449332 ms total) 2024-02-06T21:50:05.373 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: 2024-02-06T21:50:05.373 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [----------] Global test environment tear-down 2024-02-06T21:50:05.373 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (449332 ms total) 2024-02-06T21:50:05.373 INFO:tasks.workunit.client.0.smithi120.stdout: write_operations: [ PASSED ] 7 tests. 2024-02-06T21:50:09.953 INFO:tasks.workunit.client.0.smithi120.stdout: ec_list: Running main() from gmock_main.cc 2024-02-06T21:50:09.954 INFO:tasks.workunit.client.0.smithi120.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-02-06T21:50:09.954 INFO:tasks.workunit.client.0.smithi120.stdout: ec_list: [----------] Global test environment set-up. 2024-02-06T21:50:09.954 INFO:tasks.workunit.client.0.smithi120.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-02-06T21:50:09.954 INFO:tasks.workunit.client.0.smithi120.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-02-06T21:50:09.954 INFO:tasks.workunit.client.0.smithi120.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (105285 ms) 2024-02-06T21:50:09.954 INFO:tasks.workunit.client.0.smithi120.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-02-06T21:50:09.954 INFO:tasks.workunit.client.0.smithi120.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (150028 ms) 2024-02-06T21:50:09.954 INFO:tasks.workunit.client.0.smithi120.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-02-06T21:50:09.954 INFO:tasks.workunit.client.0.smithi120.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (198617 ms) 2024-02-06T21:50:09.954 INFO:tasks.workunit.client.0.smithi120.stdout: ec_list: [----------] 3 tests from NeoradosECList (453930 ms total) 2024-02-06T21:50:09.954 INFO:tasks.workunit.client.0.smithi120.stdout: ec_list: 2024-02-06T21:50:09.954 INFO:tasks.workunit.client.0.smithi120.stdout: ec_list: [----------] Global test environment tear-down 2024-02-06T21:50:09.954 INFO:tasks.workunit.client.0.smithi120.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (453930 ms total) 2024-02-06T21:50:09.954 INFO:tasks.workunit.client.0.smithi120.stdout: ec_list: [ PASSED ] 3 tests. 2024-02-06T21:50:15.910 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:09:55.724 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:15.913 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:09:55.728 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:15.973 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:09:55.788 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:15.977 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:09:55.792 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:15.979 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:09:55.743 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:15.981 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:09:55.745 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:16.047 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:09:55.811 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:16.054 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:09:55.869 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:16.072 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:09:55.887 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:36.194 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:10:16.008 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:36.197 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:10:16.012 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:36.259 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:10:16.074 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:36.262 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:10:16.077 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:36.264 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:10:16.028 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:36.268 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:10:16.082 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:36.271 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:10:16.035 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:36.344 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:10:16.108 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:36.369 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:10:16.184 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:53.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-02-06T21:50:53.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-02-06T21:50:53.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-02-06T21:50:53.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-02-06T21:50:53.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-02-06T21:50:53.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-02-06T21:50:53.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: 2024-02-06T21:50:53.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-02-06T21:50:53.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: seed 40770 2024-02-06T21:50:53.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-02-06T21:50:53.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (73 ms) 2024-02-06T21:50:53.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-02-06T21:50:53.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (100146 ms) 2024-02-06T21:50:53.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-02-06T21:50:53.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (15282 ms) 2024-02-06T21:50:53.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-02-06T21:50:53.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (38853 ms) 2024-02-06T21:50:53.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-02-06T21:50:53.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (12686 ms) 2024-02-06T21:50:53.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-02-06T21:50:53.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (17268 ms) 2024-02-06T21:50:53.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-02-06T21:50:53.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (12665 ms) 2024-02-06T21:50:53.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-02-06T21:50:53.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (5345 ms) 2024-02-06T21:50:53.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-02-06T21:50:53.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (75 ms) 2024-02-06T21:50:53.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-02-06T21:50:53.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (12082 ms) 2024-02-06T21:50:53.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-02-06T21:50:53.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (91 ms) 2024-02-06T21:50:53.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-02-06T21:50:53.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (159 ms) 2024-02-06T21:50:53.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-02-06T21:50:53.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (1829 ms) 2024-02-06T21:50:53.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-02-06T21:50:53.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: osd_max_attr_size = 0 2024-02-06T21:50:53.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-02-06T21:50:53.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (183897 ms) 2024-02-06T21:50:53.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-02-06T21:50:53.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (8829 ms) 2024-02-06T21:50:53.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-02-06T21:50:53.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-02-06T21:50:53.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: done waiting, doing copies 2024-02-06T21:50:53.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: waiting for final deep scrubs... 2024-02-06T21:50:53.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: done waiting 2024-02-06T21:50:53.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (76019 ms) 2024-02-06T21:50:53.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-02-06T21:50:53.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (83 ms) 2024-02-06T21:50:53.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-02-06T21:50:53.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (42 ms) 2024-02-06T21:50:53.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-02-06T21:50:53.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4913 ms) 2024-02-06T21:50:53.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-02-06T21:50:53.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-02-06T21:50:53.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-02-06T21:50:53.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-02-06T21:50:53.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-02-06T21:50:53.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-02-06T21:50:56.299 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:10:36.114 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:56.305 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:10:36.120 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:56.351 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:10:36.114 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:56.354 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:10:36.118 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:56.373 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:10:36.188 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:56.378 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:10:36.192 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:56.378 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:10:36.193 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:56.404 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:10:36.169 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:50:56.459 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:10:36.274 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:10.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [-pleWrite 2024-02-06T21:51:10.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (8185 ms) 2024-02-06T21:51:10.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-02-06T21:51:10.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (8000 ms) 2024-02-06T21:51:10.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-02-06T21:51:10.558 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7502 ms) 2024-02-06T21:51:10.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-02-06T21:51:10.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7277 ms) 2024-02-06T21:51:10.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-02-06T21:51:10.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (9675 ms) 2024-02-06T21:51:10.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-02-06T21:51:10.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (8078 ms) 2024-02-06T21:51:10.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-02-06T21:51:10.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6844 ms) 2024-02-06T21:51:10.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-02-06T21:51:10.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7933 ms) 2024-02-06T21:51:10.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-02-06T21:51:10.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (8208 ms) 2024-02-06T21:51:10.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-02-06T21:51:10.559 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7337 ms) 2024-02-06T21:51:10.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-02-06T21:51:10.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6989 ms) 2024-02-06T21:51:10.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-02-06T21:51:10.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (8493 ms) 2024-02-06T21:51:10.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-02-06T21:51:10.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (8035 ms) 2024-02-06T21:51:10.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-02-06T21:51:10.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7721 ms) 2024-02-06T21:51:10.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-02-06T21:51:10.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7096 ms) 2024-02-06T21:51:10.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-02-06T21:51:10.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7785 ms) 2024-02-06T21:51:10.560 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (125158 ms total) 2024-02-06T21:51:10.561 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: 2024-02-06T21:51:10.561 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [----------] Global test environment tear-down 2024-02-06T21:51:10.561 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (515039 ms total) 2024-02-06T21:51:10.561 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio: [ PASSED ] 42 tests. 2024-02-06T21:51:16.232 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:10:56.046 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:16.239 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:10:56.054 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:16.303 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:10:56.067 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:16.304 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:10:56.118 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:16.311 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:10:56.126 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:16.317 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:10:56.081 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:16.328 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:10:56.142 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:16.403 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:10:56.167 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:16.410 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:10:56.225 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:25.965 INFO:tasks.workunit.client.0.smithi120.stdout:---------] 22 tests from LibRadosMiscPP (490338 ms total) 2024-02-06T21:51:25.965 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: 2024-02-06T21:51:25.965 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-02-06T21:51:25.965 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-02-06T21:51:25.965 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (3249 ms) 2024-02-06T21:51:25.965 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (3249 ms total) 2024-02-06T21:51:25.965 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: 2024-02-06T21:51:25.965 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-02-06T21:51:25.966 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-02-06T21:51:25.966 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (410 ms) 2024-02-06T21:51:25.966 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-02-06T21:51:25.966 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (61 ms) 2024-02-06T21:51:25.966 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (471 ms total) 2024-02-06T21:51:25.966 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: 2024-02-06T21:51:25.966 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-02-06T21:51:25.966 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-02-06T21:51:25.966 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (366 ms) 2024-02-06T21:51:25.966 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-02-06T21:51:25.966 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (235 ms) 2024-02-06T21:51:25.966 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (601 ms total) 2024-02-06T21:51:25.966 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: 2024-02-06T21:51:25.966 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-02-06T21:51:25.967 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-02-06T21:51:25.967 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (976 ms) 2024-02-06T21:51:25.967 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-02-06T21:51:25.967 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (44 ms) 2024-02-06T21:51:25.967 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (1020 ms total) 2024-02-06T21:51:25.967 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: 2024-02-06T21:51:25.967 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-02-06T21:51:25.967 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-02-06T21:51:25.967 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1943 ms) 2024-02-06T21:51:25.967 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1943 ms total) 2024-02-06T21:51:25.967 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: 2024-02-06T21:51:25.967 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-02-06T21:51:25.968 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (530304 ms total) 2024-02-06T21:51:25.968 INFO:tasks.workunit.client.0.smithi120.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-02-06T21:51:36.311 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:11:16.125 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:36.314 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:11:16.128 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:36.368 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:11:16.132 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:36.370 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:11:16.185 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:36.374 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:11:16.139 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:36.378 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:11:16.193 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:36.383 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:11:16.198 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:36.419 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:11:16.184 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:36.427 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:11:16.242 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:46.985 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:11:26.800 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:46.988 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:11:26.803 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:47.020 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:11:26.835 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:47.038 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:11:26.802 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:47.040 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:11:26.804 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:47.043 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:11:26.858 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:47.046 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:11:26.861 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:47.076 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:11:26.840 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:47.083 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:11:26.898 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:56.537 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:11:36.352 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:56.543 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:11:36.358 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:56.577 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:11:36.392 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:56.601 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:11:36.365 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:56.603 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:11:36.367 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:56.608 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:11:36.423 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:56.612 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:11:36.426 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:56.632 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:11:36.396 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:51:56.638 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:11:36.453 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:16.548 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:11:56.362 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:16.550 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:11:56.365 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:16.593 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:11:56.407 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:16.600 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:11:56.414 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:16.603 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:11:56.418 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:16.605 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:11:56.369 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:16.610 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:11:56.374 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:16.646 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:11:56.461 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:16.662 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:11:56.426 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:31.757 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:12:11.571 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:31.759 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:12:11.574 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:31.792 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:12:11.606 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:31.808 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:12:11.623 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:31.811 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:12:11.626 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:31.812 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:12:11.576 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:31.813 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:12:11.578 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:31.849 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:12:11.664 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:31.869 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:12:11.633 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:36.833 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:12:16.648 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:36.843 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:12:16.658 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:36.875 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:12:16.690 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:36.904 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:12:16.668 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:36.911 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:12:16.726 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:36.911 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:12:16.675 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:36.918 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:12:16.733 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:36.931 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:12:16.695 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:36.945 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:12:16.760 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:46.873 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:12:26.687 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:46.881 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:12:26.696 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:46.924 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:12:26.739 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:46.927 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:12:26.691 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:46.933 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:12:26.698 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:46.949 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:12:26.764 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:46.956 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:12:26.771 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:46.960 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:12:26.724 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:52:46.986 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:12:26.801 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:01.750 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:12:41.564 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:01.754 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:12:41.569 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:01.784 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:12:41.598 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:01.803 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:12:41.568 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:01.805 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:12:41.569 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:01.813 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:12:41.628 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:01.815 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:12:41.630 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:01.832 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:12:41.597 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:01.847 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:12:41.662 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:02.606 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:12:42.370 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:02.607 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:12:42.371 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:02.616 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:12:42.380 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:02.621 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:12:42.436 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:02.622 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:12:42.437 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:02.635 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:12:42.450 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:02.639 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:12:42.454 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:02.640 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:12:42.455 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:02.653 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:12:42.467 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:21.991 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:01.805 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:21.993 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:01.808 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:22.021 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:01.835 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:22.052 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:01.867 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:22.056 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:01.820 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:22.057 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:01.821 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:22.058 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:01.872 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:22.083 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:01.847 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:22.085 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:01.899 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:37.119 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:16.934 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:37.127 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:16.942 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:37.176 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:16.991 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:37.199 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:16.963 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:37.201 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:16.965 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:37.201 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:17.016 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:37.204 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:17.019 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:37.247 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:17.011 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:37.247 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:17.062 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:38.158 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:17.922 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:38.159 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:17.923 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:38.168 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:17.932 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:38.193 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:18.007 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:38.194 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:18.009 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:38.210 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:18.025 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:38.213 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:18.028 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:38.214 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:18.029 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:38.225 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:18.040 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:42.265 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:22.079 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:42.271 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:22.086 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:42.290 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:22.105 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:42.323 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:22.087 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:42.330 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:22.094 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:42.336 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:22.151 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:42.341 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:22.156 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:42.346 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:22.110 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:42.364 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:22.178 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:52.191 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:32.005 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:52.195 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:32.010 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:52.220 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:32.035 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:52.242 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:32.007 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:52.243 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:32.058 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:52.246 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:32.010 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:52.247 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:32.062 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:52.271 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:32.035 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:53:52.273 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:32.088 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:02.331 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:42.145 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:02.334 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:42.149 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:02.375 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:42.189 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:02.392 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:42.207 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:02.396 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:42.211 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:02.403 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:42.167 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:02.412 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:42.177 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:02.424 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:42.239 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:02.441 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:42.205 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:07.471 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:47.285 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:07.478 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:47.293 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:07.501 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:13:47.316 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:07.540 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:47.304 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:07.546 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:47.361 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:07.546 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:47.311 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:07.548 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:47.363 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:07.569 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:13:47.384 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:07.576 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:13:47.341 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:22.373 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:02.136 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:22.376 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:02.141 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:22.426 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:02.241 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:22.434 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:02.249 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:22.449 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:02.264 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:22.452 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:02.267 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:22.456 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:02.220 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:22.473 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:02.288 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:22.547 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:02.361 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:27.566 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-02-06T21:54:27.566 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-02-06T21:54:27.566 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-02-06T21:54:27.566 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: seed 40702 2024-02-06T21:54:27.566 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-02-06T21:54:27.566 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (101792 ms) 2024-02-06T21:54:27.567 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-02-06T21:54:27.567 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (86858 ms) 2024-02-06T21:54:27.567 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-02-06T21:54:27.567 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (9124 ms) 2024-02-06T21:54:27.567 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (197774 ms total) 2024-02-06T21:54:27.567 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: 2024-02-06T21:54:27.567 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-02-06T21:54:27.567 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-02-06T21:54:27.567 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (85976 ms) 2024-02-06T21:54:27.567 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-02-06T21:54:27.567 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (46604 ms) 2024-02-06T21:54:27.568 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-02-06T21:54:27.568 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (21771 ms) 2024-02-06T21:54:27.568 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-02-06T21:54:27.568 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: my_snaps [3] 2024-02-06T21:54:27.568 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: my_snaps [4,3] 2024-02-06T21:54:27.568 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: my_snaps [5,4,3] 2024-02-06T21:54:27.568 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-02-06T21:54:27.568 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: promoting some heads 2024-02-06T21:54:27.568 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: promoting from clones for snap 6 2024-02-06T21:54:27.568 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: promoting from clones for snap 5 2024-02-06T21:54:27.569 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: promoting from clones for snap 4 2024-02-06T21:54:27.569 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: promoting from clones for snap 3 2024-02-06T21:54:27.569 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: waiting for scrubs... 2024-02-06T21:54:27.569 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: done waiting 2024-02-06T21:54:27.569 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (146192 ms) 2024-02-06T21:54:27.569 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-02-06T21:54:27.569 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (12063 ms) 2024-02-06T21:54:27.569 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-02-06T21:54:27.569 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (10297 ms) 2024-02-06T21:54:27.569 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-02-06T21:54:27.569 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (11296 ms) 2024-02-06T21:54:27.569 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-02-06T21:54:27.570 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (11036 ms) 2024-02-06T21:54:27.570 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-02-06T21:54:27.570 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (13541 ms) 2024-02-06T21:54:27.570 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-02-06T21:54:27.570 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8945 ms) 2024-02-06T21:54:27.570 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-02-06T21:54:27.570 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (11347 ms) 2024-02-06T21:54:27.570 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-02-06T21:54:27.570 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (40012 ms) 2024-02-06T21:54:27.570 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-02-06T21:54:27.570 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (9625 ms) 2024-02-06T21:54:27.571 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-02-06T21:54:27.571 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (11845 ms) 2024-02-06T21:54:27.571 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-02-06T21:54:27.571 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (16525 ms) 2024-02-06T21:54:27.571 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-02-06T21:54:27.571 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (10090 ms) 2024-02-06T21:54:27.571 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-02-06T21:54:27.571 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (12734 ms) 2024-02-06T21:54:42.420 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:22.235 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:42.423 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:22.238 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:42.462 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:22.277 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:42.469 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:22.233 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:42.470 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:22.234 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:42.478 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:22.293 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:42.480 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:22.295 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:42.508 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:22.273 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:42.520 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:22.335 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:47.581 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:27.396 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:47.586 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:27.401 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:47.601 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:27.416 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:47.634 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:27.399 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:47.636 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:27.401 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:47.643 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:27.458 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:47.645 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:27.460 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:47.655 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:27.419 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:47.659 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:27.474 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:49.419 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-02-06T21:54:49.419 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: hmm, no HitSet yet 2024-02-06T21:54:49.419 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: ok, hit_set contains 216:602f83fe:::foo:head 2024-02-06T21:54:49.419 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9215 ms) 2024-02-06T21:54:49.419 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-02-06T21:54:49.419 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg_num = 32 2024-02-06T21:54:49.419 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 0 ls 1707256484,0 2024-02-06T21:54:49.419 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 1 ls 1707256484,0 2024-02-06T21:54:49.420 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 2 ls 1707256484,0 2024-02-06T21:54:49.420 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 3 ls 1707256484,0 2024-02-06T21:54:49.420 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 4 ls 1707256484,0 2024-02-06T21:54:49.420 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 5 ls 1707256484,0 2024-02-06T21:54:49.420 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 6 ls 1707256484,0 2024-02-06T21:54:49.420 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 7 ls 1707256484,0 2024-02-06T21:54:49.420 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 8 ls 1707256484,0 2024-02-06T21:54:49.420 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 9 ls 1707256484,0 2024-02-06T21:54:49.420 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 10 ls 1707256484,0 2024-02-06T21:54:49.420 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 11 ls 1707256484,0 2024-02-06T21:54:49.420 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 12 ls 1707256484,0 2024-02-06T21:54:49.420 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 13 ls 1707256484,0 2024-02-06T21:54:49.421 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 14 ls 1707256484,0 2024-02-06T21:54:49.421 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 15 ls 1707256484,0 2024-02-06T21:54:49.421 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 16 ls 1707256484,0 2024-02-06T21:54:49.421 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 17 ls 1707256484,0 2024-02-06T21:54:49.421 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 18 ls 1707256484,0 2024-02-06T21:54:49.421 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 19 ls 1707256484,0 2024-02-06T21:54:49.421 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 20 ls 1707256484,0 2024-02-06T21:54:49.421 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 21 ls 1707256484,0 2024-02-06T21:54:49.421 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 22 ls 1707256484,0 2024-02-06T21:54:49.421 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 23 ls 1707256484,0 2024-02-06T21:54:49.421 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 24 ls 1707256484,0 2024-02-06T21:54:49.421 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 25 ls 1707256484,0 2024-02-06T21:54:49.421 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 26 ls 1707256484,0 2024-02-06T21:54:49.422 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 27 ls 1707256484,0 2024-02-06T21:54:49.422 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 28 ls 1707256484,0 2024-02-06T21:54:49.422 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 29 ls 1707256485,0 2024-02-06T21:54:49.422 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 30 ls 1707256484,0 2024-02-06T21:54:49.422 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg 31 ls 1707256484,0 2024-02-06T21:54:49.422 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: pg_num = 32 2024-02-06T21:54:49.422 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:6cac518f:::0:head 2024-02-06T21:54:49.422 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:02547ec2:::1:head 2024-02-06T21:54:49.422 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:f905c69b:::2:head 2024-02-06T21:54:49.422 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:cfc208b3:::3:head 2024-02-06T21:54:49.422 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:d83876eb:::4:head 2024-02-06T21:54:49.422 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:b29083e3:::5:head 2024-02-06T21:54:49.422 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:c4fdafeb:::6:head 2024-02-06T21:54:49.423 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:5c6b0b28:::7:head 2024-02-06T21:54:49.423 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:bd63b0f1:::8:head 2024-02-06T21:54:49.423 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:e960b815:::9:head 2024-02-06T21:54:49.423 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:52ea6a34:::10:head 2024-02-06T21:54:49.423 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:89d3ae78:::11:head 2024-02-06T21:54:49.423 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:de5d7c5f:::12:head 2024-02-06T21:54:49.423 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:566253c9:::13:head 2024-02-06T21:54:49.423 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:62a1935d:::14:head 2024-02-06T21:54:49.423 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:863748b0:::15:head 2024-02-06T21:54:49.423 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:3958e169:::16:head 2024-02-06T21:54:49.423 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:4d4dabf9:::17:head 2024-02-06T21:54:49.423 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:8391935d:::18:head 2024-02-06T21:54:49.423 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:28883081:::19:head 2024-02-06T21:54:49.424 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:69259c59:::20:head 2024-02-06T21:54:49.424 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:4bdb80b7:::21:head 2024-02-06T21:54:49.424 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:a11c5d71:::22:head 2024-02-06T21:54:49.424 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:271af37b:::23:head 2024-02-06T21:54:49.424 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:95b121be:::24:head 2024-02-06T21:54:49.424 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:58d1031b:::25:head 2024-02-06T21:54:49.424 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:0a050783:::26:head 2024-02-06T21:54:49.424 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:c709704c:::27:head 2024-02-06T21:54:49.424 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:cbe56eaf:::28:head 2024-02-06T21:54:49.424 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:86b4b162:::29:head 2024-02-06T21:54:49.424 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:70d89383:::30:head 2024-02-06T21:54:49.424 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:dd450c7c:::31:head 2024-02-06T21:54:49.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:6d5729b1:::32:head 2024-02-06T21:54:49.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:c388f3fb:::33:head 2024-02-06T21:54:49.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:56cfea31:::34:head 2024-02-06T21:54:49.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:9dbc1bf7:::35:head 2024-02-06T21:54:49.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:40b74ccd:::36:head 2024-02-06T21:54:49.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:4d5aaf42:::37:head 2024-02-06T21:54:49.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:920f362c:::38:head 2024-02-06T21:54:49.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:6cc53222:::39:head 2024-02-06T21:54:49.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:9cad833f:::40:head 2024-02-06T21:54:49.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:1ea84d41:::41:head 2024-02-06T21:54:49.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:c4480ef6:::42:head 2024-02-06T21:54:49.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:a694361e:::43:head 2024-02-06T21:54:49.425 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:d1bd33e9:::44:head 2024-02-06T21:54:49.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:ddc2cd5d:::45:head 2024-02-06T21:54:49.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:2b782207:::46:head 2024-02-06T21:54:49.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:7b187fca:::47:head 2024-02-06T21:54:49.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:90ecdf6f:::48:head 2024-02-06T21:54:49.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:a5ed95fe:::49:head 2024-02-06T21:54:49.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:ea0eaa55:::50:head 2024-02-06T21:54:49.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:f33ef17b:::51:head 2024-02-06T21:54:49.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:a0d1b2f6:::52:head 2024-02-06T21:54:49.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:60c5229e:::53:head 2024-02-06T21:54:49.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:edcbc575:::54:head 2024-02-06T21:54:49.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:102cf253:::55:head 2024-02-06T21:54:49.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:efb7fb0b:::56:head 2024-02-06T21:54:49.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:50d0a326:::57:head 2024-02-06T21:54:49.426 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:d4dc5daf:::58:head 2024-02-06T21:54:49.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:3a130462:::59:head 2024-02-06T21:54:49.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:ec87ed71:::60:head 2024-02-06T21:54:49.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:d5bc9454:::61:head 2024-02-06T21:54:49.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:3ddfe313:::62:head 2024-02-06T21:54:49.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:7c2816b9:::63:head 2024-02-06T21:54:49.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:47e00e4d:::64:head 2024-02-06T21:54:49.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:c6410c18:::65:head 2024-02-06T21:54:49.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:b48ed237:::66:head 2024-02-06T21:54:49.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:cd63ad31:::67:head 2024-02-06T21:54:49.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:b179e92b:::68:head 2024-02-06T21:54:49.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:0d9f741a:::69:head 2024-02-06T21:54:49.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:6d3352ae:::70:head 2024-02-06T21:54:49.427 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:c6d5c19e:::71:head 2024-02-06T21:54:49.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:bc4729c3:::72:head 2024-02-06T21:54:49.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:77e930b9:::73:head 2024-02-06T21:54:49.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:0abeecfd:::74:head 2024-02-06T21:54:49.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:b7c37e15:::75:head 2024-02-06T21:54:49.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:b6378398:::76:head 2024-02-06T21:54:49.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:02bd68de:::77:head 2024-02-06T21:54:49.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:cc795d2d:::78:head 2024-02-06T21:54:49.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:630d4fea:::79:head 2024-02-06T21:54:49.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:e0d29ef5:::80:head 2024-02-06T21:54:49.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:fd6f13d2:::81:head 2024-02-06T21:54:49.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:606461d5:::82:head 2024-02-06T21:54:49.428 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:eadbdc43:::83:head 2024-02-06T21:54:49.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:8761d0bb:::84:head 2024-02-06T21:54:49.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:9ef0186f:::85:head 2024-02-06T21:54:49.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:e0d41294:::86:head 2024-02-06T21:54:49.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:961de695:::87:head 2024-02-06T21:54:49.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:1423148f:::88:head 2024-02-06T21:54:49.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:633a8fa2:::89:head 2024-02-06T21:54:49.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:a8653809:::90:head 2024-02-06T21:54:49.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:3dac8b33:::91:head 2024-02-06T21:54:49.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:35aad435:::92:head 2024-02-06T21:54:49.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:f6dcc343:::93:head 2024-02-06T21:54:49.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:dbbdad87:::94:head 2024-02-06T21:54:49.429 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:1cb48ce0:::95:head 2024-02-06T21:54:49.430 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:03cd461c:::96:head 2024-02-06T21:54:49.430 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:17a4ea99:::97:head 2024-02-06T21:54:49.430 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:9993c9a7:::98:head 2024-02-06T21:54:49.430 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:6394211c:::99:head 2024-02-06T21:54:49.430 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:94c7ae57:::100:head 2024-02-06T21:54:49.430 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:6fdee5bb:::101:head 2024-02-06T21:54:49.430 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:9a477fd1:::102:head 2024-02-06T21:54:49.430 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:eb850916:::103:head 2024-02-06T21:54:49.430 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:affc56b9:::104:head 2024-02-06T21:54:49.430 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:b42dc814:::105:head 2024-02-06T21:54:49.430 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:f319f8f0:::106:head 2024-02-06T21:54:49.430 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:9a40b9de:::107:head 2024-02-06T21:54:49.431 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:8b524f28:::108:head 2024-02-06T21:54:49.431 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:e3de589f:::109:head 2024-02-06T21:54:49.431 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:90f90a5b:::110:head 2024-02-06T21:54:49.431 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:a7b4f1d7:::111:head 2024-02-06T21:54:49.431 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:af51766e:::112:head 2024-02-06T21:54:49.431 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:b6f90bd1:::113:head 2024-02-06T21:54:49.431 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:e0261208:::114:head 2024-02-06T21:54:49.431 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:c9569ef7:::115:head 2024-02-06T21:54:49.431 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:61bebe50:::116:head 2024-02-06T21:54:49.431 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:fe93412b:::117:head 2024-02-06T21:54:49.431 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:d3d38bee:::118:head 2024-02-06T21:54:49.431 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:3100ba0c:::119:head 2024-02-06T21:54:49.431 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:d0560ada:::120:head 2024-02-06T21:54:49.431 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:f0ea8b35:::121:head 2024-02-06T21:54:49.432 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:766f231a:::122:head 2024-02-06T21:54:49.432 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:a07a2582:::123:head 2024-02-06T21:54:49.432 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:bd7c6b3a:::124:head 2024-02-06T21:54:49.432 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:fb2ddaff:::125:head 2024-02-06T21:54:49.432 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:4408e1fe:::126:head 2024-02-06T21:54:49.432 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:ee1df7a7:::127:head 2024-02-06T21:54:49.432 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:c3002909:::128:head 2024-02-06T21:54:49.432 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:4f48ffa9:::129:head 2024-02-06T21:54:49.432 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:edf38733:::130:head 2024-02-06T21:54:49.432 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:c08425c0:::131:head 2024-02-06T21:54:49.432 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:5f902d98:::132:head 2024-02-06T21:54:49.432 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:41ea2c93:::133:head 2024-02-06T21:54:49.432 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:813cee13:::134:head 2024-02-06T21:54:49.432 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:0131818d:::135:head 2024-02-06T21:54:49.433 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:26ba5a85:::136:head 2024-02-06T21:54:49.433 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:381b8a5a:::137:head 2024-02-06T21:54:49.433 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:28797e47:::138:head 2024-02-06T21:54:49.433 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:bfca7f22:::139:head 2024-02-06T21:54:49.433 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:36807075:::140:head 2024-02-06T21:54:49.433 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:80b03975:::141:head 2024-02-06T21:54:49.433 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:5c15709b:::142:head 2024-02-06T21:54:49.433 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:f39ea15e:::143:head 2024-02-06T21:54:49.433 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:ea992956:::144:head 2024-02-06T21:54:49.433 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:48887b1c:::145:head 2024-02-06T21:54:49.433 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:9f24a9dd:::146:head 2024-02-06T21:54:49.433 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:987f100b:::147:head 2024-02-06T21:54:49.433 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:d2dd3581:::148:head 2024-02-06T21:54:49.433 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:7fed1808:::149:head 2024-02-06T21:54:49.434 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:c80b70e9:::150:head 2024-02-06T21:54:49.434 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:85ed90f9:::151:head 2024-02-06T21:54:49.434 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:36428b24:::152:head 2024-02-06T21:54:49.434 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:d044c34a:::153:head 2024-02-06T21:54:49.434 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:7c18bf58:::154:head 2024-02-06T21:54:49.434 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:d1c21232:::155:head 2024-02-06T21:54:49.434 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:a7a3c575:::156:head 2024-02-06T21:54:49.434 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:87da0633:::157:head 2024-02-06T21:54:49.434 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:d5ac3822:::158:head 2024-02-06T21:54:49.434 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:3f20522d:::159:head 2024-02-06T21:54:49.434 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:6ca26563:::160:head 2024-02-06T21:54:49.434 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:532ce135:::161:head 2024-02-06T21:54:49.434 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:c78863e6:::162:head 2024-02-06T21:54:57.626 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:37.440 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:57.629 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:37.443 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:57.646 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:37.461 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:57.680 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:37.444 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:57.681 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:37.445 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:57.687 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:37.502 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:57.688 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:37.503 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:57.699 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:37.463 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:54:57.703 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:37.518 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:07.514 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:47.328 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:07.516 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:47.331 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:07.542 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:47.357 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:07.562 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:47.326 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:07.564 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:47.329 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:07.568 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:47.382 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:07.569 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:47.384 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:07.590 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:47.355 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:07.594 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:47.409 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:12.531 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:52.344 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:12.535 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:52.349 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:12.549 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:14:52.364 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:12.576 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:52.340 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:12.578 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:52.342 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:12.582 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:52.397 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:12.587 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:52.402 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:12.591 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:14:52.355 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:12.600 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:14:52.414 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:22.596 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:02.360 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:22.597 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:02.362 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:22.604 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:02.419 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:22.609 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:02.424 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:22.618 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:02.383 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:22.651 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:02.466 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:22.690 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:02.505 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:22.695 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:02.510 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:22.725 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:02.540 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:27.620 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:07.435 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:27.672 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:07.487 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:27.684 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:07.448 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:27.690 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:07.454 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:27.694 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:07.509 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:27.706 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:07.470 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:27.723 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:07.538 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:27.726 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:07.541 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:27.742 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:07.557 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:32.752 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:12.567 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:32.755 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:12.570 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:32.784 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:12.599 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:32.801 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:12.565 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:32.802 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:12.566 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:32.815 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:12.630 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:32.816 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:12.631 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:32.818 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:12.582 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:32.832 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:12.647 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:47.727 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:27.541 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:47.728 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:27.543 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:47.764 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:27.579 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:47.770 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:27.534 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:47.774 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:27.538 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:47.787 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:27.602 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:47.788 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:27.603 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:47.809 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:27.574 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:47.821 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:27.636 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:48.302 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:28.066 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:48.303 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:28.067 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:48.311 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:28.075 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:48.412 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:28.226 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:48.413 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:28.228 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:48.419 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:28.233 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:48.420 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:28.235 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:48.423 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:28.238 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:48.428 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:28.243 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:58.893 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:38.708 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:58.896 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:38.711 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:58.917 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:38.732 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:58.947 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:38.762 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:58.948 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:38.763 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:58.950 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:38.715 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:58.952 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:38.716 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:58.969 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:38.784 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:55:58.973 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:38.737 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:01.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:7462ddf6:::163:head 2024-02-06T21:56:01.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:5d165639:::164:head 2024-02-06T21:56:01.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:f43765fc:::165:head 2024-02-06T21:56:01.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:b4c720e9:::166:head 2024-02-06T21:56:01.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:e694b040:::167:head 2024-02-06T21:56:01.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:afa38db2:::168:head 2024-02-06T21:56:01.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:77ba9f53:::169:head 2024-02-06T21:56:01.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:87495034:::170:head 2024-02-06T21:56:01.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:7c96bf0e:::171:head 2024-02-06T21:56:01.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:dbe346cc:::172:head 2024-02-06T21:56:01.729 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:e943ec24:::173:head 2024-02-06T21:56:01.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:f97a9c0c:::174:head 2024-02-06T21:56:01.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:6f26e74d:::175:head 2024-02-06T21:56:01.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:4f95e106:::176:head 2024-02-06T21:56:01.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:0e6f2f8f:::177:head 2024-02-06T21:56:01.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:05db05f1:::178:head 2024-02-06T21:56:01.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:38a78d66:::179:head 2024-02-06T21:56:01.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:d095610b:::180:head 2024-02-06T21:56:01.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:a1a9d709:::181:head 2024-02-06T21:56:01.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:1e5d39db:::182:head 2024-02-06T21:56:01.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:f7df4fb9:::183:head 2024-02-06T21:56:01.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:03a7f161:::184:head 2024-02-06T21:56:01.730 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:ba70721e:::185:head 2024-02-06T21:56:01.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:28e5662d:::186:head 2024-02-06T21:56:01.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:973d52de:::187:head 2024-02-06T21:56:01.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:4303eb1c:::188:head 2024-02-06T21:56:01.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:b990b48e:::189:head 2024-02-06T21:56:01.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:29b8165b:::190:head 2024-02-06T21:56:01.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:3547f197:::191:head 2024-02-06T21:56:01.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:7e260936:::192:head 2024-02-06T21:56:01.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:1abec7b1:::193:head 2024-02-06T21:56:01.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:10fdda93:::194:head 2024-02-06T21:56:01.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:15817eea:::195:head 2024-02-06T21:56:01.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:770bab57:::196:head 2024-02-06T21:56:01.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:ed9e13e7:::197:head 2024-02-06T21:56:01.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:71471a8f:::198:head 2024-02-06T21:56:01.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: checking for 217:10fb1d02:::199:head 2024-02-06T21:56:01.731 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (22916 ms) 2024-02-06T21:56:01.732 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-02-06T21:56:01.732 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707256507,0 2024-02-06T21:56:01.732 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: first is 1707256507 2024-02-06T21:56:01.732 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707256507,0 2024-02-06T21:56:01.732 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707256507,0 2024-02-06T21:56:01.732 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707256507,1707256510,1707256511,0 2024-02-06T21:56:01.732 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707256507,1707256510,1707256511,0 2024-02-06T21:56:01.732 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707256507,1707256510,1707256511,0 2024-02-06T21:56:01.732 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707256507,1707256510,1707256511,1707256513,1707256514,0 2024-02-06T21:56:01.732 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707256507,1707256510,1707256511,1707256513,1707256514,0 2024-02-06T21:56:01.732 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707256507,1707256510,1707256511,1707256513,1707256514,0 2024-02-06T21:56:01.732 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707256507,1707256510,1707256511,1707256513,1707256514,1707256516,1707256517,0 2024-02-06T21:56:01.732 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707256507,1707256510,1707256511,1707256513,1707256514,1707256516,1707256517,0 2024-02-06T21:56:01.733 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707256507,1707256510,1707256511,1707256513,1707256514,1707256516,1707256517,0 2024-02-06T21:56:01.733 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707256511,1707256513,1707256514,1707256516,1707256517,1707256519,1707256520,0 2024-02-06T21:56:01.733 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: first now 1707256511, trimmed 2024-02-06T21:56:01.733 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22496 ms) 2024-02-06T21:56:01.733 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-02-06T21:56:01.733 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: foo0 2024-02-06T21:56:01.733 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-02-06T21:56:01.733 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (17762 ms) 2024-02-06T21:56:01.733 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-02-06T21:56:07.739 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:47.554 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:07.785 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:47.549 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:07.786 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:47.551 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:07.789 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:47.604 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:07.813 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:47.577 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:07.816 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:47.631 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:07.861 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:47.676 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:07.864 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:47.679 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:07.900 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:47.715 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:08.401 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:48.165 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:08.401 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:48.166 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:08.416 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:15:48.180 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:08.574 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:48.389 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:08.575 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:48.390 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:08.585 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:15:48.400 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:08.589 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:48.404 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:08.590 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:48.405 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:08.603 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:15:48.417 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:22.855 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:02.669 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:22.859 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:02.674 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:22.888 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:02.702 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:22.905 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:02.720 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:22.906 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:02.670 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:22.907 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:02.671 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:22.909 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:02.724 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:22.932 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:02.696 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:22.945 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:02.760 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:28.024 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:07.839 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:28.032 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:07.846 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:28.054 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:07.869 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:28.090 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:07.854 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:28.092 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:07.856 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:28.095 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:07.910 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:28.098 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:07.913 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:28.107 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:07.871 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:28.116 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:07.931 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:38.079 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:17.894 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:38.137 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:17.952 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:38.166 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:17.981 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:38.202 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:18.017 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:38.203 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:17.967 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:38.204 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:18.019 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:38.205 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:17.969 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:38.243 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:18.058 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:38.251 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:18.015 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:48.167 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:27.981 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:48.177 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:27.992 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:48.208 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:28.023 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:48.231 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:27.995 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:48.231 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:28.046 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:48.235 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:28.000 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:48.239 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:28.054 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:48.268 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:28.033 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:48.275 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:28.089 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:53.201 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:32.965 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:53.202 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:32.966 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:53.216 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:33.030 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:53.219 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:33.033 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:53.220 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:32.984 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:53.242 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:33.056 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:53.270 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:33.085 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:53.276 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:33.091 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:56:53.297 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:33.112 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:08.263 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:48.077 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:08.267 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:48.082 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:08.297 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:16:48.112 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:08.319 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:48.084 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:08.325 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:48.089 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:08.326 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:48.140 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:08.329 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:48.144 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:08.357 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:16:48.121 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:08.363 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:16:48.178 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.213 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:08.028 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.216 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:08.031 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.242 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:08.057 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.261 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:08.025 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.264 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:08.028 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.266 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:08.081 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.268 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:08.083 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.288 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:08.052 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.294 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:08.109 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.933 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:08.697 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.934 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:08.698 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.942 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:08.706 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.954 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:08.769 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.956 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:08.771 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.965 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:08.780 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.995 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:08.810 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:28.996 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:08.811 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:29.006 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:08.821 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:43.456 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:23.270 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:43.504 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:23.268 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:43.505 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:23.269 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:43.524 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:23.339 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:43.535 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:23.299 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:43.562 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:23.377 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:43.586 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:23.401 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:43.587 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:23.402 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:43.615 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:23.430 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:48.485 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:28.300 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:48.488 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:28.303 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:48.513 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:28.328 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:48.535 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:28.299 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:48.539 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:28.304 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:48.552 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:28.367 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:48.554 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:28.318 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:48.556 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:28.371 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:48.575 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:28.390 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:58.443 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:38.257 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:58.443 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:38.207 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:58.447 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:38.262 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:58.452 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:38.216 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:58.485 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:38.300 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:58.504 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:38.268 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:58.521 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:38.336 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:58.528 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:38.343 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:57:58.565 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:38.379 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:08.448 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:48.262 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:08.451 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:48.266 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:08.477 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:48.292 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:08.495 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:48.259 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:08.498 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:48.313 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:08.499 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:48.264 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:08.501 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:48.316 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:08.524 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:48.288 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:08.535 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:48.350 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:13.458 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:53.273 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:13.463 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:53.278 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:13.475 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:53.290 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:13.506 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:53.271 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:13.510 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:53.274 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:13.512 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:53.327 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:13.519 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:53.334 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:13.521 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:53.285 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:13.530 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:53.345 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:15.715 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:55.527 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:15.716 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:55.530 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:15.730 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:17:55.545 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:15.763 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:55.527 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:15.764 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:55.528 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:15.775 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:55.590 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:15.776 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:55.591 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:15.779 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:17:55.543 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:15.794 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:17:55.609 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:33.705 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:18:13.520 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:33.707 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:18:13.522 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:33.733 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:18:13.548 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:33.754 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:18:13.518 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:33.755 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:18:13.520 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:33.757 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:18:13.572 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:33.759 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:18:13.574 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:33.780 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:18:13.544 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:33.785 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:18:13.600 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:53.601 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:18:33.415 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:53.605 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:18:33.420 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:53.639 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:18:33.453 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:53.652 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:18:33.416 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:53.657 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:18:33.421 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:53.664 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:18:33.479 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:53.666 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:18:33.481 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:53.688 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:18:33.452 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:53.712 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:18:33.526 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:54.376 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:18:34.140 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:54.377 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:18:34.142 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:54.387 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:18:34.151 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:54.513 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:18:34.328 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:54.514 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:18:34.329 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:54.524 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:18:34.339 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:54.587 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:18:34.401 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:54.588 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:18:34.403 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:58:54.597 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:18:34.412 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:08.598 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:18:48.413 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:08.647 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:18:48.462 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:08.675 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:18:48.489 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:08.697 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:18:48.461 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:08.701 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:18:48.466 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:08.709 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:18:48.524 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:08.710 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:18:48.525 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:08.728 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:18:48.492 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:08.739 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:18:48.554 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:13.654 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:18:53.469 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:13.658 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:18:53.473 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:13.672 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:18:53.487 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:13.700 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:18:53.464 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:13.703 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:18:53.467 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:13.706 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:18:53.521 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:13.709 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:18:53.524 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:13.715 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:18:53.479 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:13.721 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:18:53.536 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:21.811 INFO:tasks.ceph.osd.7.smithi135.stderr:2024-02-06T21:59:21.796+0000 21e7e640 -1 snap_mapper.add_oid found existing snaps mapped on 235:1ef2636f:test-rados-api-smithi120-40702-80::foo:2, removing 2024-02-06T21:59:21.819 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T21:59:21.811+0000 2c693640 -1 snap_mapper.add_oid found existing snaps mapped on 235:1ef2636f:test-rados-api-smithi120-40702-80::foo:2, removing 2024-02-06T21:59:28.728 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:08.542 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:28.729 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:08.544 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:28.763 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:08.578 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:28.778 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:08.592 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:28.779 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:08.544 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:28.781 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:08.596 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:28.785 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:08.550 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:28.806 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:08.621 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:28.810 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:08.574 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:33.748 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:13.563 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:33.752 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:13.567 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:33.769 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:13.584 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:33.799 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:13.614 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:33.801 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:13.565 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:33.801 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:13.616 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:33.804 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:13.568 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:33.818 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:13.633 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:33.821 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:13.586 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:43.874 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:23.688 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:43.876 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:23.691 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:43.918 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:23.733 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:43.927 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:23.692 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:43.936 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:23.701 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:43.940 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:23.755 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:43.942 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:23.757 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:43.967 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:23.732 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:43.969 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:23.784 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:53.832 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:33.646 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:53.835 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:33.651 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:53.863 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:33.678 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:53.885 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:33.649 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:53.888 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:33.652 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:53.893 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:33.708 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:53.896 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:33.711 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:53.913 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:33.677 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:53.923 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:33.738 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:56.586 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (21774 ms) 2024-02-06T21:59:56.586 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-02-06T21:59:56.586 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (25930 ms) 2024-02-06T21:59:56.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-02-06T21:59:56.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (4173 ms) 2024-02-06T21:59:56.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-02-06T21:59:56.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T21:59:56.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (5585 ms) 2024-02-06T21:59:56.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-02-06T21:59:56.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4253 ms) 2024-02-06T21:59:56.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-02-06T21:59:56.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T21:59:56.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (5573 ms) 2024-02-06T21:59:56.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-02-06T21:59:56.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T21:59:56.587 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5062 ms) 2024-02-06T21:59:56.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-02-06T21:59:56.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T21:59:56.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40226 ms) 2024-02-06T21:59:56.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-02-06T21:59:56.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T21:59:56.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17979 ms) 2024-02-06T21:59:56.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-02-06T21:59:56.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T21:59:56.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5484 ms) 2024-02-06T21:59:56.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-02-06T21:59:56.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T21:59:56.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4659 ms) 2024-02-06T21:59:56.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-02-06T21:59:56.588 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T21:59:56.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (27096 ms) 2024-02-06T21:59:56.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-02-06T21:59:56.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T21:59:56.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (16457 ms) 2024-02-06T21:59:56.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-02-06T21:59:56.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T21:59:56.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6471 ms) 2024-02-06T21:59:56.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-02-06T21:59:56.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T21:59:56.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5691 ms) 2024-02-06T21:59:56.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-02-06T21:59:56.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T21:59:56.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: waiting for scrubs... 2024-02-06T21:59:56.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: done waiting 2024-02-06T21:59:56.589 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26735 ms) 2024-02-06T21:59:56.590 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-02-06T21:59:56.590 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T21:59:56.590 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (11668 ms) 2024-02-06T21:59:56.590 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-02-06T21:59:56.590 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T21:59:56.590 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (11163 ms) 2024-02-06T21:59:56.590 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-02-06T21:59:56.590 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T21:59:56.590 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (10652 ms) 2024-02-06T21:59:59.027 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:38.842 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:59.034 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:38.849 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:59.057 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:38.872 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:59.087 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:38.901 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:59.091 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:38.906 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:59.096 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:38.860 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:59.099 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:38.863 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:59.112 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:38.927 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T21:59:59.118 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:38.882 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:13.971 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:53.786 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:13.973 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:53.788 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.006 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:53.821 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.025 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:53.789 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.026 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:53.841 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.027 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:53.791 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.028 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:53.843 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.060 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:53.824 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.064 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:53.879 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.797 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:54.561 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.798 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:54.562 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.806 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:19:54.570 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.821 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:54.636 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.822 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:54.637 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.830 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:19:54.645 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.894 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:54.709 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.895 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:54.710 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:14.903 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:19:54.718 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:28.609 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:08.423 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:28.613 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:08.428 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:28.650 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:08.465 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:28.668 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:08.483 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:28.670 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:08.485 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:28.671 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:08.435 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:28.674 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:08.438 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:28.709 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:08.524 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:28.711 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:08.475 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:29.391 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:09.155 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:29.392 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:09.156 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:29.401 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:09.165 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:29.459 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:09.273 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:29.459 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:09.275 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:29.473 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:09.288 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:29.474 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:09.289 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:29.475 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:09.290 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:29.484 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:09.299 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:34.072 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:13.887 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:34.077 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:13.892 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:34.093 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:13.908 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:34.120 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:13.884 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:34.127 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:13.888 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:34.128 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:13.942 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:34.130 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:13.945 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:34.135 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:13.899 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:34.145 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:13.960 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:49.236 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:29.051 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:49.240 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:29.055 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:49.245 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:29.009 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:49.245 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:29.060 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:49.291 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:29.106 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:49.297 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:29.061 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:49.310 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:29.125 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:49.316 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:29.131 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:49.327 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:29.092 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:54.209 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:34.024 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:54.212 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:34.027 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:54.232 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:34.047 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:54.260 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:34.074 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:54.261 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:34.025 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:54.263 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:34.028 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:54.265 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:34.080 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:54.278 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:34.042 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:00:54.284 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:34.099 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:04.275 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:44.090 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:04.276 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:44.091 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:04.293 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:44.108 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:04.322 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:44.085 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:04.322 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:44.087 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:04.325 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:44.140 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:04.327 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:44.142 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:04.339 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:44.103 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:04.343 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:44.158 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:05.648 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T22:01:05.639+0000 29e8e640 -1 snap_mapper.add_oid found existing snaps mapped on 111:44925ee8:test-rados-api-smithi120-40702-97::foo:21, removing 2024-02-06T22:01:05.681 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T22:01:05.637+0000 21e7e640 -1 snap_mapper.add_oid found existing snaps mapped on 111:44925ee8:test-rados-api-smithi120-40702-97::foo:21, removing 2024-02-06T22:01:14.320 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:54.135 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:14.325 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:54.140 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:14.361 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:54.176 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:14.369 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:54.134 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:14.372 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:54.136 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:14.377 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:54.192 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:14.380 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:54.195 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:14.397 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:54.162 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:14.406 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:54.221 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:19.453 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:59.268 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:19.455 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:59.270 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:19.472 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:20:59.287 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:19.504 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:59.319 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:19.506 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:59.321 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:19.515 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:59.279 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:19.516 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:59.281 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:19.530 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:20:59.345 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:19.532 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:20:59.297 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:34.406 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:14.221 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:34.476 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:14.290 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:34.530 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:14.294 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:34.534 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:14.298 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:34.534 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:14.349 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:34.548 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:14.363 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:34.554 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:14.369 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:34.569 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:14.333 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:34.611 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:14.426 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:35.575 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:15.339 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:35.576 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:15.340 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:35.587 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:15.351 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:35.776 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:15.591 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:35.777 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:15.592 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:35.793 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:15.608 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:35.953 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:15.768 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:35.954 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:15.769 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:35.965 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:15.780 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:44.584 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:24.399 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:44.641 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:24.456 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:44.674 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:24.489 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:44.700 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:24.465 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:44.702 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:24.517 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:44.705 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:24.520 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:44.723 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:24.487 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:44.736 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:24.551 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:44.754 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:24.517 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:49.627 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:29.441 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:49.629 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:29.444 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:49.647 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:29.462 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:49.675 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:29.439 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:49.677 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:29.492 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:49.677 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:29.442 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:49.680 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:29.495 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:49.694 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:29.458 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:49.697 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:29.512 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:54.827 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:34.642 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:54.876 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:34.691 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:54.910 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:34.725 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:54.919 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:34.683 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:54.922 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:34.686 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:54.941 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:34.756 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:54.943 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:34.707 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:54.947 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:34.762 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:01:54.970 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:34.785 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:04.776 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:44.591 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:04.779 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:44.594 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:04.814 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:44.629 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:04.831 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:44.646 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:04.836 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:44.651 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:04.837 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:44.601 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:04.842 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:44.606 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:04.867 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:44.682 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:04.877 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:44.642 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:14.956 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:54.770 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:14.960 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:54.775 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:14.987 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:54.803 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:15.008 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:54.772 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:15.008 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:54.823 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:15.009 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:54.774 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:15.011 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:54.826 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:15.036 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:54.800 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:15.046 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:54.862 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:19.901 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:59.716 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:19.905 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:59.720 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:19.925 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:21:59.740 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:19.965 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:59.780 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:19.969 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:59.734 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:19.970 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:59.785 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:19.974 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:59.738 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:19.988 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:21:59.753 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:19.991 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:21:59.806 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:28.554 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:08.369 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:28.559 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:08.374 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:28.607 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:08.421 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:28.609 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:08.424 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:28.637 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:08.401 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:28.638 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:08.403 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:28.649 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:08.464 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:28.677 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:08.442 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:28.690 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:08.505 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:35.030 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:14.845 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:35.076 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:14.841 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:35.077 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:14.842 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:35.096 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:14.861 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:35.115 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:14.929 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:35.151 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:14.966 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:35.165 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:14.980 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:35.168 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:14.983 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:35.187 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:15.002 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:50.057 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:29.872 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:50.062 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:29.826 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:50.062 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:29.877 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:50.064 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:29.828 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:50.095 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:29.859 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:50.129 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:29.943 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:50.134 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:29.949 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:50.146 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:29.961 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:50.170 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:29.985 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:55.182 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:34.997 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:55.185 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:35.000 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:55.185 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:35.000 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:55.189 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:35.004 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:55.200 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:35.015 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:55.202 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:35.017 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:55.232 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:34.997 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:55.234 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:34.999 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:02:55.248 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:35.012 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:10.166 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:49.980 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:10.169 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:49.984 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:10.190 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:50.005 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:10.216 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:49.980 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:10.217 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:50.032 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:10.218 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:49.982 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:10.220 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:50.035 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:10.236 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:50.001 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:10.246 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:50.061 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:15.313 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:55.128 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:15.381 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:55.196 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:15.398 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:22:55.213 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:15.444 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:55.208 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:15.446 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:55.210 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:15.462 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:55.277 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:15.466 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:55.281 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:15.467 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:22:55.231 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:15.492 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:22:55.307 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:25.209 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:05.024 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:25.213 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:05.028 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:25.241 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:05.056 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:25.265 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:05.029 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:25.266 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:05.030 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:25.278 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:05.092 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:25.280 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:05.095 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:25.292 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:05.057 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:25.308 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:05.123 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:35.244 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:15.058 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:35.246 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:15.061 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:35.274 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:15.089 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:35.296 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:15.111 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:35.298 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:15.113 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:35.313 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:15.078 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:35.315 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:15.079 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:35.325 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:15.140 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:35.342 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:15.106 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:40.253 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:20.068 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:40.257 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:20.072 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:40.275 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:20.090 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:40.300 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:20.065 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:40.304 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:20.119 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:40.305 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:20.069 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:40.308 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:20.123 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:40.320 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:20.084 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:40.323 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:20.138 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:55.311 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:35.125 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:55.315 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:35.130 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:55.352 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:35.167 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:55.365 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:35.180 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:55.369 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:35.184 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:55.373 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:35.137 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:55.375 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:35.139 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:55.404 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:35.219 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:55.413 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:35.177 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:56.076 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:35.840 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:56.077 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:35.841 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:56.085 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:35.850 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:56.165 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:35.980 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:56.166 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:35.981 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:56.174 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:35.989 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:56.224 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:36.039 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:56.225 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:36.040 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:03:56.237 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:36.052 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:10.155 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-02-06T22:04:10.155 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T22:04:10.155 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (10900 ms) 2024-02-06T22:04:10.156 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-02-06T22:04:10.156 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T22:04:10.156 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (8160 ms) 2024-02-06T22:04:10.156 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-02-06T22:04:10.156 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T22:04:10.156 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6954 ms) 2024-02-06T22:04:10.156 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-02-06T22:04:10.156 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T22:04:10.156 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (28856 ms) 2024-02-06T22:04:10.156 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-02-06T22:04:10.156 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T22:04:10.156 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15337 ms) 2024-02-06T22:04:10.156 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-02-06T22:04:10.156 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (14373 ms) 2024-02-06T22:04:10.157 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-02-06T22:04:10.157 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-02-06T22:04:10.157 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: 00000010 69 73 |is| 2024-02-06T22:04:10.157 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: 00000012 2024-02-06T22:04:10.157 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (11735 ms) 2024-02-06T22:04:10.157 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-02-06T22:04:10.157 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: require_osd_release = squid 2024-02-06T22:04:10.157 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (7914 ms) 2024-02-06T22:04:10.157 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (913150 ms total) 2024-02-06T22:04:10.157 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: 2024-02-06T22:04:10.157 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-02-06T22:04:10.157 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-02-06T22:04:10.157 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1651 ms) 2024-02-06T22:04:10.157 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-02-06T22:04:10.158 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (13029 ms) 2024-02-06T22:04:10.158 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-02-06T22:04:10.158 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (19583 ms) 2024-02-06T22:04:10.158 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-02-06T22:04:10.158 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (66 ms) 2024-02-06T22:04:10.158 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (34329 ms total) 2024-02-06T22:04:10.158 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: 2024-02-06T22:04:10.158 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-02-06T22:04:10.158 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-02-06T22:04:10.158 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (8175 ms) 2024-02-06T22:04:10.158 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-02-06T22:04:10.158 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (9503 ms) 2024-02-06T22:04:10.158 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-02-06T22:04:10.158 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: waiting for scrub... 2024-02-06T22:04:10.158 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: done waiting 2024-02-06T22:04:10.159 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (29086 ms) 2024-02-06T22:04:10.159 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-02-06T22:04:10.159 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (12159 ms) 2024-02-06T22:04:10.159 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-02-06T22:04:10.159 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (9843 ms) 2024-02-06T22:04:10.159 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-02-06T22:04:10.159 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (11393 ms) 2024-02-06T22:04:10.159 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-02-06T22:04:10.159 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (12813 ms) 2024-02-06T22:04:10.159 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-02-06T22:04:10.365 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:50.180 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:10.368 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:50.183 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:10.369 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:50.133 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:10.371 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:50.135 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:10.411 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:50.226 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:10.412 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:50.175 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:10.425 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:50.240 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:10.427 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:50.242 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:10.464 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:50.279 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:15.378 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:55.193 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:15.383 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:55.198 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:15.401 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:23:55.216 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:15.431 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:55.246 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:15.433 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:55.197 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:15.434 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:55.249 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:15.438 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:55.202 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:15.456 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:23:55.221 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:15.457 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:23:55.272 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:25.370 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:05.185 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:25.419 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:05.233 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:25.440 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:05.255 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:25.464 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:05.278 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:25.464 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:05.228 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:25.467 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:05.231 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:25.469 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:05.284 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:25.484 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:05.248 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:25.489 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:05.304 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:31.059 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:10.874 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:31.064 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:10.879 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:31.096 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:10.911 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:31.123 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:10.938 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:31.125 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:10.940 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:31.127 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:10.891 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:31.130 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:10.894 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:31.148 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:10.963 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:31.157 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:10.922 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:35.442 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:15.257 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:35.444 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:15.259 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:35.457 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:15.272 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:35.489 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:15.304 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:35.492 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:15.256 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:35.492 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:15.307 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:35.498 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:15.262 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:35.513 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:15.328 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:35.515 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:15.280 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:40.528 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:20.343 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:40.574 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:20.339 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:40.576 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:20.340 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:40.586 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:20.401 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:40.592 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:20.356 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:40.606 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:20.421 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:40.649 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:20.464 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:40.651 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:20.466 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:40.676 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:20.491 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:55.479 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:35.294 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:55.534 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:35.298 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:55.535 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:35.350 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:55.542 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:35.306 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:55.580 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:35.344 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:55.592 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:35.406 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:55.597 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:35.412 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:55.602 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:35.417 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:55.646 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:35.461 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:56.173 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:35.937 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:56.174 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:35.938 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:56.182 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:35.946 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:56.412 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:36.227 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:56.413 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:36.229 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:56.426 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:36.241 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:56.589 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:36.404 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:56.590 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:36.405 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:04:56.599 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:36.414 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:10.815 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:50.629 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:10.821 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:50.636 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:10.862 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:50.677 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:10.869 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:50.633 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:10.872 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:50.636 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:10.880 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:50.695 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:10.882 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:50.697 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:10.911 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:50.675 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:10.934 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:50.749 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:15.591 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:55.406 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:15.641 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:55.456 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:15.658 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:24:55.473 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:15.686 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:55.450 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:15.689 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:55.453 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:15.692 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:55.507 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:15.696 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:55.511 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:15.704 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:24:55.468 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:15.711 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:24:55.526 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:25.692 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:05.507 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:25.696 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:05.511 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:25.725 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:05.540 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:25.743 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:05.507 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:25.746 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:05.510 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:25.758 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:05.573 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:25.759 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:05.574 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:25.772 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:05.536 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:25.787 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:05.601 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:35.731 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:15.545 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:35.735 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:15.550 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:35.756 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:15.571 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:35.778 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:15.542 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:35.780 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:15.595 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:35.781 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:15.545 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:35.783 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:15.598 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:35.800 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:15.564 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:35.801 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:15.616 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:50.832 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:30.646 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:50.836 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:30.651 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:50.878 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:30.693 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:50.886 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:30.651 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:50.888 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:30.652 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:50.888 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:30.703 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:50.892 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:30.707 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:50.915 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:30.679 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:50.921 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:30.736 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:55.802 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:35.617 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:55.850 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:35.665 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:55.868 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:35.683 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:55.896 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:35.660 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:55.898 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:35.662 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:55.902 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:35.717 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:55.903 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:35.718 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:55.915 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:35.679 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:05:55.919 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:35.734 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:05.875 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:45.690 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:05.879 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:45.694 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:05.906 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:45.721 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:05.923 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:45.687 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:05.925 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:45.690 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:05.936 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:45.751 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:05.939 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:45.754 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:05.950 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:45.714 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:05.966 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:45.781 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:15.899 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:55.714 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:15.901 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:55.716 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:15.922 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:25:55.737 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:15.943 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:55.708 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:15.949 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:55.713 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:15.951 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:55.766 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:15.955 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:55.770 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:15.967 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:25:55.731 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:15.972 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:25:55.787 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:21.026 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:00.841 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:21.031 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:00.846 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:21.048 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:00.863 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:21.077 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:00.841 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:21.077 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:00.842 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:21.090 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:00.854 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:21.092 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:00.907 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:21.093 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:00.908 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:21.112 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:00.927 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:36.173 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:15.988 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:36.178 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:15.993 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:36.227 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:15.991 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:36.230 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:15.994 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:36.238 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:16.053 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:36.241 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:16.056 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:36.252 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:16.067 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:36.268 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:16.032 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:36.284 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:16.099 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:51.162 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:30.977 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:51.165 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:30.980 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:51.197 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:31.012 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:51.227 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:31.042 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:51.230 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:31.045 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:51.235 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:30.999 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:51.240 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:31.004 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:51.263 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:31.078 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:51.268 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:31.032 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:56.280 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:36.094 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:56.287 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:36.102 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:56.315 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:36.130 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:56.343 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:36.158 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:56.399 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:36.160 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:56.399 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:36.184 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:56.399 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:36.121 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:56.399 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:36.124 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:06:56.399 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:36.145 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:06.138 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:45.953 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:06.148 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:45.963 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:06.207 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:45.971 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:06.214 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:46.029 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:06.216 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:46.031 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:06.217 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:45.981 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:06.219 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:46.034 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:06.244 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:46.008 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:06.253 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:46.068 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:11.233 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:51.048 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:11.239 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:51.054 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:11.265 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:51.080 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:11.290 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:51.055 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:11.293 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:51.058 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:11.299 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:51.114 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:11.301 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:51.116 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:11.314 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:51.078 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:11.324 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:51.139 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:11.914 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (11016 ms) 2024-02-06T22:07:11.914 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-02-06T22:07:11.914 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (16321 ms) 2024-02-06T22:07:11.914 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-02-06T22:07:11.915 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (10825 ms) 2024-02-06T22:07:11.915 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-02-06T22:07:11.915 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (15489 ms) 2024-02-06T22:07:11.915 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-02-06T22:07:11.915 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (9183 ms) 2024-02-06T22:07:11.915 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-02-06T22:07:11.915 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8770 ms) 2024-02-06T22:07:11.915 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-02-06T22:07:11.915 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: hmm, no HitSet yet 2024-02-06T22:07:11.915 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: ok, hit_set contains 267:602f83fe:::foo:head 2024-02-06T22:07:11.915 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (10009 ms) 2024-02-06T22:07:11.915 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-02-06T22:07:11.915 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707257129,0 2024-02-06T22:07:11.916 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: first is 1707257129 2024-02-06T22:07:11.916 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707257129,0 2024-02-06T22:07:11.916 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707257129,0 2024-02-06T22:07:11.916 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707257129,1707257132,1707257133,0 2024-02-06T22:07:11.916 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707257129,1707257132,1707257133,0 2024-02-06T22:07:11.916 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707257129,1707257132,1707257133,0 2024-02-06T22:07:11.916 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707257129,1707257132,1707257133,1707257135,1707257136,0 2024-02-06T22:07:11.916 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707257129,1707257132,1707257133,1707257135,1707257136,0 2024-02-06T22:07:11.916 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707257129,1707257132,1707257133,1707257135,1707257136,0 2024-02-06T22:07:11.916 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707257129,1707257132,1707257133,1707257135,1707257136,1707257138,1707257139,0 2024-02-06T22:07:11.916 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707257129,1707257132,1707257133,1707257135,1707257136,1707257138,1707257139,0 2024-02-06T22:07:11.916 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707257129,1707257132,1707257133,1707257135,1707257136,1707257138,1707257139,0 2024-02-06T22:07:11.917 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: got ls 1707257133,1707257135,1707257136,1707257138,1707257139,1707257141,1707257142,0 2024-02-06T22:07:11.917 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: first now 1707257133, trimmed 2024-02-06T22:07:11.917 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22739 ms) 2024-02-06T22:07:11.917 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-02-06T22:07:11.917 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: foo0 2024-02-06T22:07:11.917 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-02-06T22:07:11.917 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (18428 ms) 2024-02-06T22:07:11.917 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-02-06T22:07:11.917 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (20589 ms) 2024-02-06T22:07:11.917 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-02-06T22:07:11.917 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (25290 ms) 2024-02-06T22:07:11.917 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-02-06T22:07:11.917 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (6821 ms) 2024-02-06T22:07:11.917 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-02-06T22:07:11.918 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4774 ms) 2024-02-06T22:07:11.918 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-02-06T22:07:11.918 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (5452 ms) 2024-02-06T22:07:11.918 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-02-06T22:07:11.918 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3520 ms) 2024-02-06T22:07:11.918 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (282199 ms total) 2024-02-06T22:07:11.918 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: 2024-02-06T22:07:11.918 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-02-06T22:07:11.918 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1476273 ms total) 2024-02-06T22:07:11.918 INFO:tasks.workunit.client.0.smithi120.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-02-06T22:07:11.920 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:07:11.920 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40442 2024-02-06T22:07:11.920 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40442 2024-02-06T22:07:11.920 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:07:11.920 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40617 2024-02-06T22:07:11.920 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40617 2024-02-06T22:07:11.920 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:07:11.920 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41105 2024-02-06T22:07:11.920 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41105 2024-02-06T22:07:11.920 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:07:11.920 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40901 2024-02-06T22:07:11.921 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40901 2024-02-06T22:07:11.921 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:07:11.921 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40757 2024-02-06T22:07:11.921 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40757 2024-02-06T22:07:11.921 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:07:11.921 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40526 2024-02-06T22:07:11.921 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40526 2024-02-06T22:07:11.921 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:07:11.921 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40825 2024-02-06T22:07:11.921 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40825 2024-02-06T22:07:16.342 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:56.157 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:16.346 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:56.161 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:16.366 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:26:56.181 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:16.389 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:56.153 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:16.393 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:56.157 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:16.394 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:56.209 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:16.398 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:56.213 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:16.405 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:26:56.170 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:16.412 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:26:56.227 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:21.256 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:01.071 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:21.262 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:01.077 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:21.275 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:01.090 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:21.308 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:01.072 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:21.309 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:01.123 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:21.310 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:01.125 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:21.312 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:01.076 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:21.323 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:01.087 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:21.324 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:01.136 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:31.321 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:11.135 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:31.322 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:11.137 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:31.340 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:11.155 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:31.366 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:11.130 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:31.369 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:11.133 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:31.371 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:11.186 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:31.373 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:11.188 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:31.384 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:11.148 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:31.394 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:11.209 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:36.324 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:16.138 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:36.328 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:16.143 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:36.340 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:16.155 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:36.373 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:16.137 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:36.375 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:16.140 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:36.376 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:16.191 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:36.378 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:16.193 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:36.386 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:16.150 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:36.388 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:16.203 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:41.355 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:21.170 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:41.360 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:21.175 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:41.372 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:21.187 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:41.406 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:21.221 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:41.406 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:21.170 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:41.408 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:21.223 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:41.409 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:21.173 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:41.421 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:21.185 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:41.422 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:21.237 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:56.392 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:36.206 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:56.397 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:36.212 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:56.416 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:36.231 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:56.448 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:36.262 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:56.448 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:36.212 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:56.449 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:36.264 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:56.451 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:36.215 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:56.466 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:36.281 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:07:56.469 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:36.234 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:01.406 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:41.221 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:01.410 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:41.224 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:01.424 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:41.239 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:01.461 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:41.225 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:01.466 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:41.230 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:01.474 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:41.289 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:01.478 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:41.293 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:01.478 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:41.242 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:01.490 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:41.305 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:16.446 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:56.261 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:16.448 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:56.263 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:16.463 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:27:56.278 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:16.499 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:56.314 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:16.500 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:56.316 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:16.502 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:56.266 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:16.505 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:56.269 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:16.519 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:27:56.334 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:16.523 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:27:56.288 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:21.456 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:01.271 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:21.460 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:01.275 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:21.476 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:01.291 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:21.503 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:01.267 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:21.511 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:01.326 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:21.515 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:01.330 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:21.517 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:01.282 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:21.528 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:01.343 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:21.533 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:01.297 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:36.490 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:16.305 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:36.495 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:16.310 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:36.513 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:16.328 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:36.544 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:16.308 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:36.547 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:16.312 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:36.558 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:16.373 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:36.561 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:16.376 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:36.564 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:16.328 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:36.578 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:16.393 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:41.559 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:21.374 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:41.563 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:21.378 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:41.576 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:21.391 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:41.606 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:21.371 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:41.608 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:21.373 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:41.611 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:21.426 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:41.613 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:21.429 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:41.628 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:21.443 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:41.636 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:21.400 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:42.269 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:22.033 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:42.270 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:22.034 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:42.278 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:22.042 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:42.293 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:22.108 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:42.294 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:22.109 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:42.307 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:22.122 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:42.313 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:22.128 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:42.314 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:22.129 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:42.322 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:22.137 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:56.587 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:36.401 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:56.590 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:36.405 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:56.607 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:36.422 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:56.637 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:36.402 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:56.638 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:36.453 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:56.640 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:36.404 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:56.641 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:36.456 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:56.656 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:36.420 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:08:56.657 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:36.472 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:01.556 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:41.371 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:01.602 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:41.417 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:01.615 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:41.430 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:01.649 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:41.413 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:01.650 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:41.465 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:01.651 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:41.415 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:01.652 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:41.467 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:01.663 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:41.427 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:01.663 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:41.478 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:16.637 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:56.452 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:16.640 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:56.455 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:16.655 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:28:56.470 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:16.686 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:56.450 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:16.689 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:56.453 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:16.693 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:56.508 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:16.694 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:56.509 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:16.706 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:28:56.470 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:16.710 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:28:56.525 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:26.685 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:06.500 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:26.687 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:06.503 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:26.706 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:06.521 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:26.731 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:06.495 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:26.734 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:06.498 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:26.736 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:06.551 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:26.739 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:06.554 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:26.747 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:06.512 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:26.754 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:06.569 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:27.396 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:07.160 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:27.397 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:07.161 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:27.409 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:07.174 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:27.431 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:07.246 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:27.431 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:07.247 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:27.449 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:07.264 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:27.456 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:07.271 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:27.457 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:07.272 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:27.497 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:07.312 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:36.687 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:16.502 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:36.688 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:16.503 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:36.703 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:16.518 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:36.733 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:16.497 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:36.735 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:16.499 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:36.737 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:16.552 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:36.740 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:16.555 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:36.747 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:16.512 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:36.753 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:16.568 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:46.715 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:26.530 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:46.716 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:26.531 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:46.729 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:26.544 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:46.760 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:26.524 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:46.762 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:26.577 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:46.763 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:26.527 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:46.765 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:26.580 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:46.776 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:26.540 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:46.779 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:26.594 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:56.734 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:36.548 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:56.737 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:36.552 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:56.759 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:36.574 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:56.786 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:36.550 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:56.787 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:36.552 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:56.792 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:36.607 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:56.795 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:36.610 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:56.800 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:36.564 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:09:56.809 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:36.624 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:06.794 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:46.609 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:06.797 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:46.612 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:06.816 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:46.631 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:06.845 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:46.660 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:06.845 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:46.609 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:06.848 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:46.663 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:06.848 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:46.613 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:06.863 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:46.628 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:06.865 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:46.680 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:07.437 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:47.201 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:07.438 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:47.202 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:07.446 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:47.210 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:07.451 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:47.267 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:07.453 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:47.268 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:07.460 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:47.276 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:07.487 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:47.302 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:07.491 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:47.306 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:07.499 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:47.315 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:16.827 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:56.641 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:16.828 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:56.643 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:16.843 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:29:56.658 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:16.874 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:56.688 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:16.877 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:56.692 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:16.879 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:56.644 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:16.882 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:56.646 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:16.895 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:29:56.710 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:16.895 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:29:56.660 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:26.808 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:06.622 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:26.853 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:06.617 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:26.856 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:06.620 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:26.857 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:06.672 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:26.873 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:06.688 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:26.873 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:06.637 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:26.907 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:06.722 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:26.908 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:06.723 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:26.920 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:06.735 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:36.876 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:16.690 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:36.880 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:16.695 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:36.894 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:16.709 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:36.927 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:16.742 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:36.927 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:16.692 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:36.929 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:16.744 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:36.929 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:16.694 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:36.942 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:16.757 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:36.943 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:16.707 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:47.020 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:26.834 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:47.021 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:26.836 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:47.036 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:26.851 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:47.062 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:26.827 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:47.063 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:26.828 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:47.077 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:26.841 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:47.088 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:26.903 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:47.089 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:26.904 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:47.103 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:26.918 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:56.936 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:36.751 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:56.940 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:36.755 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:56.959 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:36.774 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:56.989 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:36.804 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:56.990 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:36.805 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:56.991 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:36.755 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:56.992 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:36.757 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:57.009 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:36.824 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:10:57.012 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:36.777 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:01.950 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:41.764 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:01.952 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:41.767 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:01.966 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:41.781 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:01.998 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:41.762 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:01.999 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:41.814 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:02.000 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:41.765 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:02.003 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:41.818 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:02.012 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:41.776 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:02.016 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:41.831 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:16.985 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:56.799 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:16.987 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:56.802 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:17.005 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:30:56.820 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:17.035 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:56.799 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:17.038 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:56.802 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:17.055 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:56.870 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:17.055 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:30:56.819 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:17.059 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:56.874 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:17.077 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:30:56.891 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:22.003 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:01.817 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:22.004 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:01.819 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:22.013 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:01.828 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:22.048 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:01.863 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:22.050 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:01.865 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:22.052 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:01.817 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:22.054 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:01.818 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:22.061 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:01.876 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:22.064 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:01.828 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:36.989 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:16.804 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:37.036 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:16.851 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:37.053 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:16.868 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:37.086 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:16.850 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:37.087 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:16.852 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:37.088 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:16.903 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:37.089 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:16.904 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:37.103 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:16.867 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:37.158 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:16.973 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:42.002 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:21.816 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:42.047 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:21.862 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:42.060 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:21.874 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:42.095 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:21.910 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:42.097 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:21.862 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:42.098 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:21.913 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:42.099 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:21.864 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:42.111 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:21.875 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:42.111 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:21.926 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:57.087 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:36.902 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:57.091 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:36.906 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:57.109 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:36.924 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:57.137 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:36.902 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:57.140 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:36.904 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:57.140 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:36.956 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:57.143 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:36.958 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:57.156 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:36.920 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:11:57.161 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:36.976 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:02.051 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:41.865 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:02.097 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:41.911 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:02.109 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:41.924 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:02.143 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:41.907 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:02.144 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:41.959 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:02.146 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:41.911 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:02.147 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:41.962 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:02.157 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:41.921 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:02.160 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:41.975 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:17.081 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:56.896 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:17.128 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:56.943 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:17.152 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:31:56.967 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:17.182 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:56.946 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:17.182 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:56.997 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:17.184 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:56.948 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:17.185 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:57.000 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:17.201 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:31:56.965 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:17.201 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:31:57.016 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:22.136 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:01.951 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:22.141 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:01.956 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:22.156 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:01.971 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:22.186 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:01.950 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:22.188 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:02.003 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:22.188 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:01.953 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:22.193 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:02.008 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:22.200 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:01.964 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:22.205 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:02.020 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.192 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:17.006 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.197 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:17.012 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.216 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:17.031 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.247 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:17.011 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.248 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:17.063 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.250 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:17.014 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.253 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:17.068 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.269 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:17.033 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.270 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:17.085 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.863 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:17.627 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.865 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:17.629 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.880 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:17.644 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.922 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:17.737 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.924 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:17.739 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.938 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:17.754 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.954 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:17.769 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.955 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:17.770 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:37.969 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:17.783 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:42.157 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:21.972 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:42.202 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:21.966 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:42.203 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:22.018 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:42.205 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:21.969 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:42.215 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:21.979 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:42.217 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:22.032 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:42.249 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:22.064 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:42.253 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:22.068 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:42.265 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:22.080 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:54.806 INFO:tasks.workunit.client.0.smithi120.stdout: ec_io: Running main() from gmock_main.cc 2024-02-06T22:12:54.807 INFO:tasks.workunit.client.0.smithi120.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-02-06T22:12:54.807 INFO:tasks.workunit.client.0.smithi120.stdout: ec_io: [----------] Global test environment set-up. 2024-02-06T22:12:54.807 INFO:tasks.workunit.client.0.smithi120.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-02-06T22:12:54.807 INFO:tasks.workunit.client.0.smithi120.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-02-06T22:12:54.807 INFO:tasks.workunit.client.0.smithi120.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (9477 ms) 2024-02-06T22:12:54.807 INFO:tasks.workunit.client.0.smithi120.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-02-06T22:12:54.807 INFO:tasks.workunit.client.0.smithi120.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (1809356 ms) 2024-02-06T22:12:54.807 INFO:tasks.workunit.client.0.smithi120.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (1818833 ms total) 2024-02-06T22:12:54.807 INFO:tasks.workunit.client.0.smithi120.stdout: ec_io: 2024-02-06T22:12:54.807 INFO:tasks.workunit.client.0.smithi120.stdout: ec_io: [----------] Global test environment tear-down 2024-02-06T22:12:54.807 INFO:tasks.workunit.client.0.smithi120.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (1818833 ms total) 2024-02-06T22:12:54.807 INFO:tasks.workunit.client.0.smithi120.stdout: ec_io: [ PASSED ] 2 tests. 2024-02-06T22:12:57.599 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:37.413 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:57.603 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:37.418 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:57.633 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:37.448 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:57.662 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:37.426 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:57.666 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:37.430 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:57.666 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:37.481 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:57.668 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:37.483 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:57.690 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:37.454 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:12:57.693 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:37.508 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:01.748 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-02-06T22:13:01.748 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-02-06T22:13:01.748 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-02-06T22:13:01.749 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-02-06T22:13:01.749 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-02-06T22:13:01.749 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: watch_notify_test_cb 2024-02-06T22:13:01.749 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (1800305 ms) 2024-02-06T22:13:01.749 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-02-06T22:13:01.749 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-02-06T22:13:01.749 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94136965965376 err -107 2024-02-06T22:13:01.749 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1098 ms) 2024-02-06T22:13:01.749 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-02-06T22:13:01.749 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-02-06T22:13:01.749 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94136965965376 err -107 2024-02-06T22:13:01.749 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1130 ms) 2024-02-06T22:13:01.750 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-02-06T22:13:01.750 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: watch_notify2_test_cb from 4436 notify_id 4526895529985 cookie 94136965965376 2024-02-06T22:13:01.750 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (96 ms) 2024-02-06T22:13:01.750 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-02-06T22:13:01.750 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: watch_notify2_test_cb from 4436 notify_id 4526895529984 cookie 94136965945488 2024-02-06T22:13:01.750 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (159 ms) 2024-02-06T22:13:01.750 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-02-06T22:13:01.750 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: watch_notify2_test_cb from 4436 notify_id 4526895529985 cookie 94136965971488 2024-02-06T22:13:01.750 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (90 ms) 2024-02-06T22:13:01.750 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-02-06T22:13:01.750 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: watch_notify2_test_cb from 4436 notify_id 4526895529984 cookie 94136965971488 2024-02-06T22:13:01.750 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: watch_notify2_test_cb from 4436 notify_id 4526895529984 cookie 94136965986176 2024-02-06T22:13:01.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (215 ms) 2024-02-06T22:13:01.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-02-06T22:13:01.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: watch_notify2_test_cb from 4436 notify_id 4526895529985 cookie 94136965986048 2024-02-06T22:13:01.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: watch_notify2_test_cb from 4436 notify_id 4526895529986 cookie 94136965986048 2024-02-06T22:13:01.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3096 ms) 2024-02-06T22:13:01.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-02-06T22:13:01.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-02-06T22:13:01.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94136965971488 err -107 2024-02-06T22:13:01.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: watch_notify2_test_cb from 4436 notify_id 4526895529985 cookie 94136965971488 2024-02-06T22:13:01.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5199 ms) 2024-02-06T22:13:01.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-02-06T22:13:01.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-02-06T22:13:01.751 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94136965971488 err -107 2024-02-06T22:13:01.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1403 ms) 2024-02-06T22:13:01.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (1812791 ms total) 2024-02-06T22:13:01.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: 2024-02-06T22:13:01.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-02-06T22:13:01.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-02-06T22:13:01.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: watch_notify_test_cb 2024-02-06T22:13:01.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1682 ms) 2024-02-06T22:13:01.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1682 ms total) 2024-02-06T22:13:01.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: 2024-02-06T22:13:01.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-02-06T22:13:01.752 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (1826023 ms total) 2024-02-06T22:13:01.753 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-02-06T22:13:02.320 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:42.135 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:02.322 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:42.137 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:02.341 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:42.156 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:02.371 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:42.135 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:02.375 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:42.140 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:02.376 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:42.191 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:02.380 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:42.195 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:02.392 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:42.156 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:02.396 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:42.212 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:04.474 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-02-06T22:13:04.474 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-02-06T22:13:04.474 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-02-06T22:13:04.475 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-02-06T22:13:04.475 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-02-06T22:13:04.475 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: notify 2024-02-06T22:13:04.475 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1808090 ms) 2024-02-06T22:13:04.475 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-02-06T22:13:04.475 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (126 ms) 2024-02-06T22:13:04.475 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1808216 ms total) 2024-02-06T22:13:04.475 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: 2024-02-06T22:13:04.475 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-02-06T22:13:04.475 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-02-06T22:13:04.475 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: notify 2024-02-06T22:13:04.475 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (551 ms) 2024-02-06T22:13:04.475 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-02-06T22:13:04.476 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: notify 2024-02-06T22:13:04.476 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (4296 ms) 2024-02-06T22:13:04.476 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-02-06T22:13:04.476 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (102 ms) 2024-02-06T22:13:04.476 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-02-06T22:13:04.476 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (95 ms) 2024-02-06T22:13:04.476 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-02-06T22:13:04.476 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: handle_notify cookie 94422082885120 notify_id 4569845202949 notifier_gid 20683 2024-02-06T22:13:04.476 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (109 ms) 2024-02-06T22:13:04.476 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-02-06T22:13:04.476 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: handle_notify cookie 94422082116592 notify_id 4569845202946 notifier_gid 20683 2024-02-06T22:13:04.476 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (143 ms) 2024-02-06T22:13:04.476 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-02-06T22:13:04.477 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: handle_notify cookie 94422082910256 notify_id 4569845202950 notifier_gid 20683 2024-02-06T22:13:04.477 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (137 ms) 2024-02-06T22:13:04.477 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-02-06T22:13:04.477 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: handle_notify cookie 94422082910256 notify_id 4569845202951 notifier_gid 20683 2024-02-06T22:13:04.477 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (129 ms) 2024-02-06T22:13:04.477 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-02-06T22:13:04.477 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: handle_notify cookie 94422082910256 notify_id 4569845202952 notifier_gid 20683 2024-02-06T22:13:04.477 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (135 ms) 2024-02-06T22:13:04.477 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-02-06T22:13:04.477 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: handle_notify cookie 94422082913536 notify_id 4569845202944 notifier_gid 20683 2024-02-06T22:13:04.477 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (226 ms) 2024-02-06T22:13:04.477 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-02-06T22:13:04.477 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: trying... 2024-02-06T22:13:04.477 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: handle_notify cookie 94422082925968 notify_id 4569845202945 notifier_gid 20683 2024-02-06T22:13:04.478 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: timed out 2024-02-06T22:13:17.365 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:57.179 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:17.365 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:57.181 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:17.395 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:57.209 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:17.408 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:57.172 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:17.409 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:57.174 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:17.411 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:57.226 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:17.414 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:57.229 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:17.444 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:57.209 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:17.448 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:57.264 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:18.132 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:57.896 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:18.135 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:57.899 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:18.153 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:32:57.917 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:18.210 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:58.025 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:18.211 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:58.026 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:18.225 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:32:58.040 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:18.253 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:58.068 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:18.254 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:58.069 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:18.264 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:32:58.079 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:20.186 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: flushing 2024-02-06T22:13:20.186 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: flushed 2024-02-06T22:13:20.186 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3106 ms) 2024-02-06T22:13:20.186 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-02-06T22:13:20.186 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: trying... 2024-02-06T22:13:20.186 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: handle_notify cookie 94422082925968 notify_id 4582730104834 notifier_gid 20683 2024-02-06T22:13:20.187 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: timed out 2024-02-06T22:13:20.187 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: flushing 2024-02-06T22:13:20.187 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: flushed 2024-02-06T22:13:20.187 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3342 ms) 2024-02-06T22:13:20.187 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-02-06T22:13:20.187 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: List watches 2024-02-06T22:13:20.187 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: notify2 2024-02-06T22:13:20.187 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: handle_notify cookie 94422082925968 notify_id 4595615006729 notifier_gid 20683 2024-02-06T22:13:20.187 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: notify2 done 2024-02-06T22:13:20.187 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: watch_check 2024-02-06T22:13:20.187 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: unwatch2 2024-02-06T22:13:20.187 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: flushing 2024-02-06T22:13:20.187 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: done 2024-02-06T22:13:20.188 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3332 ms) 2024-02-06T22:13:20.188 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-02-06T22:13:20.188 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: List watches 2024-02-06T22:13:20.188 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: notify2 2024-02-06T22:13:20.188 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: handle_notify cookie 94422082913520 notify_id 4608499908618 notifier_gid 20683 2024-02-06T22:13:20.188 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: notify2 done 2024-02-06T22:13:20.188 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: watch_check 2024-02-06T22:13:20.188 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: unwatch2 2024-02-06T22:13:20.188 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: flushing 2024-02-06T22:13:20.188 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: done 2024-02-06T22:13:20.188 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3146 ms) 2024-02-06T22:13:20.188 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (18849 ms total) 2024-02-06T22:13:20.188 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: 2024-02-06T22:13:20.188 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-02-06T22:13:20.189 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (1844452 ms total) 2024-02-06T22:13:20.189 INFO:tasks.workunit.client.0.smithi120.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-02-06T22:13:20.190 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:13:20.190 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41260 2024-02-06T22:13:20.190 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41260 2024-02-06T22:13:20.190 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:13:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41333 2024-02-06T22:13:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41333 2024-02-06T22:13:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:13:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40545 2024-02-06T22:13:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40545 2024-02-06T22:13:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:13:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41140 2024-02-06T22:13:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41140 2024-02-06T22:13:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:13:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41029 2024-02-06T22:13:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41029 2024-02-06T22:13:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:13:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40417 2024-02-06T22:13:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40417 2024-02-06T22:13:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:13:20.192 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40479 2024-02-06T22:13:20.192 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40479 2024-02-06T22:13:20.192 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:13:20.192 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41077 2024-02-06T22:13:20.192 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41077 2024-02-06T22:13:20.192 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:13:20.192 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40408 2024-02-06T22:13:20.192 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40408 2024-02-06T22:13:32.574 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:12.389 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:32.640 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:12.404 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:32.643 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:12.456 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:32.647 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:12.412 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:32.681 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:12.445 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:32.695 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:12.510 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:32.717 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:12.532 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:32.725 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:12.540 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:32.783 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:12.598 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:37.500 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:17.315 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:37.506 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:17.321 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:37.523 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:17.338 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:37.553 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:17.368 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:37.554 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:17.318 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:37.555 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:17.370 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:37.557 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:17.322 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:37.573 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:17.388 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:37.578 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:17.343 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:38.197 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:18.012 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:38.199 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:18.013 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:38.217 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:18.032 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:38.284 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:18.048 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:38.285 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:18.100 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:38.287 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:18.102 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:38.297 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:18.061 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:38.309 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:18.073 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:38.349 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:18.164 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:47.536 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:27.351 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:47.538 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:27.353 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:47.559 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:27.374 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:47.583 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:27.346 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:47.584 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:27.349 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:47.587 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:27.402 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:47.589 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:27.405 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:47.604 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:27.368 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:47.612 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:27.427 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:57.727 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:37.541 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:57.731 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:37.546 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:57.760 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:37.574 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:57.793 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:37.558 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:57.795 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:37.560 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:57.804 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:37.619 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:57.805 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:37.620 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:57.832 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:37.596 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:13:57.847 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:37.662 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:02.586 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:42.400 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:02.590 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:42.405 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:02.606 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:42.421 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:02.636 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:42.400 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:02.638 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:42.403 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:02.640 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:42.455 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:02.641 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:42.457 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:02.653 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:42.417 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:02.660 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:42.475 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:17.640 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:57.455 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:17.645 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:57.460 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:17.675 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:57.490 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:17.692 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:57.456 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:17.692 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:57.508 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:17.696 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:57.511 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:17.697 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:57.461 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:17.725 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:57.489 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:17.726 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:57.541 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:18.537 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:58.301 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:18.538 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:58.302 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:18.555 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:33:58.319 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:18.565 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:58.380 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:18.566 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:58.381 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:18.587 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:33:58.402 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:18.723 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:58.538 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:18.729 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:58.544 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:18.741 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:33:58.556 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:29.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-02-06T22:14:29.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-02-06T22:14:29.901 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-02-06T22:14:29.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-02-06T22:14:29.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-02-06T22:14:29.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (6617 ms) 2024-02-06T22:14:29.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-02-06T22:14:29.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (1822834 ms) 2024-02-06T22:14:29.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-02-06T22:14:29.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6755 ms) 2024-02-06T22:14:29.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-02-06T22:14:29.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2684 ms) 2024-02-06T22:14:29.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-02-06T22:14:29.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3691 ms) 2024-02-06T22:14:29.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-02-06T22:14:29.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3192 ms) 2024-02-06T22:14:29.902 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3338 ms) 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3152 ms) 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3344 ms) 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2622 ms) 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3249 ms) 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3545 ms) 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3279 ms) 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3200 ms) 2024-02-06T22:14:29.903 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-02-06T22:14:29.904 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3499 ms) 2024-02-06T22:14:29.904 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-02-06T22:14:29.904 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3338 ms) 2024-02-06T22:14:29.904 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-02-06T22:14:29.904 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3362 ms) 2024-02-06T22:14:29.904 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-02-06T22:14:29.904 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2644 ms) 2024-02-06T22:14:29.904 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-02-06T22:14:29.904 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3328 ms) 2024-02-06T22:14:29.904 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-02-06T22:14:29.904 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3510 ms) 2024-02-06T22:14:29.904 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-02-06T22:14:29.904 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3392 ms) 2024-02-06T22:14:29.904 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-02-06T22:14:29.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2595 ms) 2024-02-06T22:14:29.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-02-06T22:14:29.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3202 ms) 2024-02-06T22:14:29.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-02-06T22:14:29.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3544 ms) 2024-02-06T22:14:29.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-02-06T22:14:29.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3608 ms) 2024-02-06T22:14:29.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-02-06T22:14:29.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3180 ms) 2024-02-06T22:14:29.905 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-02-06T22:14:32.688 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:12.503 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:32.693 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:12.508 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:32.723 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:12.538 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:32.736 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:12.500 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:32.740 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:12.555 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:32.741 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:12.505 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:32.744 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:12.559 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:32.766 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:12.531 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:32.771 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:12.586 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:35.753 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:15.567 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:35.756 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:15.571 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:35.758 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:15.522 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:35.768 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:15.583 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:35.782 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:15.547 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:35.785 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:15.600 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:35.808 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:15.572 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:35.836 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:15.651 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:35.866 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:15.681 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:37.660 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:17.475 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:37.708 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:17.523 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:37.725 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:17.540 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:37.753 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:17.518 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:37.766 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:17.581 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:37.770 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:17.585 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:37.775 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:17.539 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:37.782 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:17.597 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:37.787 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:17.551 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:47.752 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:27.566 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:47.755 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:27.570 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:47.782 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:27.597 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:47.802 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:27.566 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:47.803 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:27.617 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:47.805 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:27.620 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:47.816 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:27.581 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:47.830 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:27.644 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:47.838 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:27.602 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:58.022 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:37.837 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:58.026 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:37.841 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:58.051 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:37.865 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:58.087 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:37.851 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:58.090 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:37.905 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:58.091 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:37.906 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:58.095 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:37.860 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:58.115 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:37.930 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:14:58.117 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:37.882 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:02.820 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:42.635 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:02.822 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:42.637 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:02.838 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:42.653 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:02.870 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:42.634 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:02.871 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:42.635 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:02.879 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:42.694 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:02.883 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:42.698 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:02.883 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:42.648 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:02.900 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:42.715 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:18.191 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:58.006 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:18.196 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:58.012 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:18.200 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:57.965 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:18.204 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:57.968 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:18.237 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:34:58.052 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:18.238 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:34:58.003 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:18.258 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:58.073 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:18.260 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:58.075 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:18.300 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:34:58.115 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:38.145 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:17.960 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:38.201 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:18.016 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:38.205 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:17.969 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:38.214 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:17.978 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:38.253 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:18.017 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:38.254 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:18.069 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:38.255 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:18.071 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:38.268 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:18.083 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:38.290 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:18.106 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:48.255 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:28.070 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:48.312 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:28.126 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:48.336 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:28.150 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:48.362 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:28.177 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:48.366 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:28.181 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:48.380 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:28.144 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:48.382 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:28.147 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:48.389 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:28.204 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:48.405 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:28.169 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:58.310 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:38.125 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:58.312 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:38.127 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:58.337 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:38.152 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:58.360 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:38.174 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:58.361 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:38.176 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:58.365 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:38.130 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:58.369 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:38.134 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:58.386 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:38.201 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:15:58.401 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:38.166 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:03.336 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:43.147 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:03.381 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:43.196 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:03.393 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:43.157 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:03.404 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:43.169 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:03.405 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:43.220 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:03.422 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:43.186 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:03.434 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:43.249 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:03.435 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:43.250 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:03.449 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:43.264 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:18.317 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:58.131 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:18.319 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:58.134 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:18.350 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:58.165 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:18.370 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:58.185 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:18.373 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:58.188 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:18.378 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:58.143 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:18.382 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:58.146 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:18.404 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:58.219 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:18.411 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:58.175 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:19.200 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:58.964 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:19.201 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:58.965 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:19.210 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:35:58.974 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:19.232 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:59.047 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:19.233 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:59.048 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:19.243 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:35:59.058 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:19.268 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:59.083 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:19.269 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:59.084 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:19.285 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:35:59.100 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:26.771 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:06.586 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:26.820 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:06.635 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:26.844 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:06.659 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:26.867 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:06.631 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:26.872 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:06.636 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:26.876 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:06.691 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:26.877 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:06.692 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:26.895 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:06.660 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:26.903 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:06.719 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:33.426 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:13.241 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:33.473 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:13.237 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:33.474 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:13.238 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:33.490 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:13.305 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:33.490 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:13.255 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:33.494 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:13.309 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:33.495 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:13.310 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:33.513 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:13.328 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:33.516 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:13.331 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:38.402 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:18.216 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:38.404 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:18.219 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:38.417 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:18.231 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:38.448 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:18.212 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:38.450 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:18.214 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:38.452 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:18.267 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:38.453 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:18.269 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:38.471 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:18.286 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:38.475 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:18.240 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:48.542 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:28.357 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:48.545 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:28.360 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:48.568 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:28.383 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:48.592 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:28.356 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:48.597 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:28.361 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:48.608 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:28.423 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:48.609 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:28.424 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:48.618 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:28.383 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:48.632 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:28.447 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:58.564 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:38.378 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:58.566 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:38.381 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:58.589 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:38.404 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:58.615 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:38.430 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:58.616 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:38.431 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:58.623 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:38.387 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:58.628 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:38.392 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:58.639 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:38.454 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:16:58.651 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:38.415 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:03.494 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:43.309 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:03.500 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:43.315 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:03.514 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:43.329 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:03.544 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:43.359 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:03.545 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:43.309 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:03.546 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:43.311 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:03.549 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:43.364 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:03.559 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:43.323 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:03.571 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:43.386 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:08.607 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripAppendPP (3679 ms) 2024-02-06T22:17:08.607 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-02-06T22:17:08.607 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3535 ms) 2024-02-06T22:17:08.607 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-02-06T22:17:08.608 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3444 ms) 2024-02-06T22:17:08.608 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-02-06T22:17:08.608 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3365 ms) 2024-02-06T22:17:08.608 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-02-06T22:17:08.608 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: setting pool EIO 2024-02-06T22:17:08.608 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: max_success 0, min_failed 1 2024-02-06T22:17:08.608 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (5786 ms) 2024-02-06T22:17:08.608 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-02-06T22:17:08.608 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3423 ms) 2024-02-06T22:17:08.608 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (1933936 ms total) 2024-02-06T22:17:08.608 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: 2024-02-06T22:17:08.608 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-02-06T22:17:08.608 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-02-06T22:17:08.608 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3539 ms) 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (10541 ms) 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (16592 ms) 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3295 ms) 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (33967 ms total) 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3355 ms) 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3355 ms total) 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-02-06T22:17:08.609 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (15665 ms) 2024-02-06T22:17:08.610 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-02-06T22:17:08.610 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7939 ms) 2024-02-06T22:17:08.610 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-02-06T22:17:08.610 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7989 ms) 2024-02-06T22:17:08.610 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-02-06T22:17:08.610 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7159 ms) 2024-02-06T22:17:08.610 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-02-06T22:17:08.610 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3316 ms) 2024-02-06T22:17:08.610 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-02-06T22:17:08.610 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7840 ms) 2024-02-06T22:17:08.610 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-02-06T22:17:08.610 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7907 ms) 2024-02-06T22:17:08.610 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-02-06T22:17:08.610 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7558 ms) 2024-02-06T22:17:08.610 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-02-06T22:17:08.611 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7054 ms) 2024-02-06T22:17:08.611 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-02-06T22:17:08.611 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7731 ms) 2024-02-06T22:17:08.611 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-02-06T22:17:08.611 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7019 ms) 2024-02-06T22:17:08.611 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-02-06T22:17:08.611 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7816 ms) 2024-02-06T22:17:08.611 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-02-06T22:17:08.611 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6838 ms) 2024-02-06T22:17:18.548 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:58.363 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:18.553 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:58.368 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:18.582 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:58.398 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:18.595 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:58.360 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:18.599 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:58.363 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:18.599 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:58.413 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:18.603 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:58.418 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:18.625 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:58.390 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:18.632 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:58.447 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:19.396 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:59.161 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:19.397 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:59.162 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:19.406 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:36:59.171 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:19.419 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:59.234 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:19.420 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:59.235 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:19.429 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:36:59.244 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:19.442 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:59.257 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:19.443 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:59.258 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:19.452 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:36:59.267 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:33.688 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:13.503 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:33.692 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:13.507 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:33.721 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:13.536 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:33.737 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:13.501 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:33.740 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:13.504 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:33.742 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:13.557 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:33.743 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:13.558 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:33.773 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:13.588 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:33.774 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:13.538 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:38.633 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:18.448 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:38.634 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:18.449 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:38.648 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:18.463 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:38.676 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:18.441 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:38.678 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:18.442 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:38.683 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:18.498 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:38.684 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:18.499 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:38.692 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:18.457 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:38.697 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:18.512 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:48.778 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:28.592 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:48.781 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:28.596 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:48.804 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:28.619 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:48.831 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:28.646 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:48.832 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:28.647 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:48.847 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:28.612 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:48.850 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:28.615 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:48.856 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:28.671 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:48.875 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:28.638 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:17:56.455 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-02-06T22:17:56.456 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3204 ms) 2024-02-06T22:17:56.456 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-02-06T22:17:56.456 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (8421 ms) 2024-02-06T22:17:56.456 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-02-06T22:17:56.456 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6757 ms) 2024-02-06T22:17:56.456 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-02-06T22:17:56.456 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6752 ms) 2024-02-06T22:17:56.456 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-02-06T22:17:56.456 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7851 ms) 2024-02-06T22:17:56.456 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-02-06T22:17:56.456 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7274 ms) 2024-02-06T22:17:56.456 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-02-06T22:17:56.457 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7586 ms) 2024-02-06T22:17:56.457 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (149676 ms total) 2024-02-06T22:17:56.457 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: 2024-02-06T22:17:56.457 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-02-06T22:17:56.457 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (2120934 ms total) 2024-02-06T22:17:56.457 INFO:tasks.workunit.client.0.smithi120.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-02-06T22:17:56.460 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.460 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40428 2024-02-06T22:17:56.460 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40428 2024-02-06T22:17:56.460 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.460 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41458 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41458 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40652 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40652 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41522 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41522 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40791 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40791 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41225 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41225 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41581 2024-02-06T22:17:56.461 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41581 2024-02-06T22:17:56.462 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.462 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40714 2024-02-06T22:17:56.462 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40714 2024-02-06T22:17:56.462 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.462 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41086 2024-02-06T22:17:56.462 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41086 2024-02-06T22:17:56.462 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.462 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41432 2024-02-06T22:17:56.462 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41432 2024-02-06T22:17:56.462 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.462 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41176 2024-02-06T22:17:56.462 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41176 2024-02-06T22:17:56.462 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.462 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41481 2024-02-06T22:17:56.462 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41481 2024-02-06T22:17:56.463 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.463 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41005 2024-02-06T22:17:56.463 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41005 2024-02-06T22:17:56.463 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.463 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40582 2024-02-06T22:17:56.463 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40582 2024-02-06T22:17:56.463 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.463 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41558 2024-02-06T22:17:56.463 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41558 2024-02-06T22:17:56.463 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.463 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40921 2024-02-06T22:17:56.463 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40921 2024-02-06T22:17:56.463 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.463 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40401 2024-02-06T22:17:56.464 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40401 2024-02-06T22:17:56.464 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.464 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40462 2024-02-06T22:17:56.464 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40462 2024-02-06T22:17:56.464 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.464 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40875 2024-02-06T22:17:56.464 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40875 2024-02-06T22:17:56.464 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.464 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40502 2024-02-06T22:17:56.464 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40502 2024-02-06T22:17:56.464 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.464 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40953 2024-02-06T22:17:56.464 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40953 2024-02-06T22:17:56.465 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.465 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=40968 2024-02-06T22:17:56.465 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 40968 2024-02-06T22:17:56.465 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.465 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41410 2024-02-06T22:17:56.465 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41410 2024-02-06T22:17:56.465 INFO:tasks.workunit.client.0.smithi120.stderr:+ for t in "${!pids[@]}" 2024-02-06T22:17:56.465 INFO:tasks.workunit.client.0.smithi120.stderr:+ pid=41625 2024-02-06T22:17:56.465 INFO:tasks.workunit.client.0.smithi120.stderr:+ wait 41625 2024-02-06T22:17:56.465 INFO:tasks.workunit.client.0.smithi120.stderr:+ exit 0 2024-02-06T22:17:56.465 INFO:tasks.workunit.client.0.smithi120.stderr:+ cleanup 2024-02-06T22:17:56.465 INFO:tasks.workunit.client.0.smithi120.stderr:+ pkill -P 40396 2024-02-06T22:17:56.535 INFO:tasks.workunit.client.0.smithi120.stderr:+ true 2024-02-06T22:17:56.535 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-06T22:17:56.535 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-06T22:17:56.565 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-02-06T22:17:56.566 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-06T22:17:56.887 DEBUG:teuthology.parallel:result is None 2024-02-06T22:17:56.888 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-02-06T22:17:56.914 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-02-06T22:17:56.914 DEBUG:teuthology.orchestra.run.smithi120:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-02-06T22:17:56.980 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-02-06T22:17:56.980 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-02-06T22:17:56.990 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-02-06T22:17:56.990 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-06T22:17:57.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.118+0000 7fc9d181e640 10 monclient: get_monmap_and_config 2024-02-06T22:17:57.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.118+0000 7fc9d181e640 10 monclient: build_initial_monmap 2024-02-06T22:17:57.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.118+0000 7fc9d181e640 10 monclient: monmap: 2024-02-06T22:17:57.120 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:17:57.120 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:57.120 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:17:57.120030+0000 2024-02-06T22:17:57.120 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:17:57.120030+0000 2024-02-06T22:17:57.120 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:17:57.121 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:57.121 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:17:57.121 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:17:57.121 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:17:57.121 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:17:57.121 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:17:57.121 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:57.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.119+0000 7fc9d181e640 1 Processor -- start 2024-02-06T22:17:57.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.119+0000 7fc9d181e640 1 -- start start 2024-02-06T22:17:57.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.119+0000 7fc9d181e640 10 monclient: init 2024-02-06T22:17:57.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.119+0000 7fc9d181e640 10 monclient: _reopen_session rank -1 2024-02-06T22:17:57.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.119+0000 7fc9d181e640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-02-06T22:17:57.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.120+0000 7fc9d181e640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc9cc0b88b0 0x7fc9cc0a99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:57.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.120+0000 7fc9d181e640 10 monclient(hunting): picked mon.noname-c con 0x7fc9cc0b88b0 addr v2:172.21.15.135:3300/0 2024-02-06T22:17:57.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.120+0000 7fc9d181e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc9cc13f8d0 addr v1:172.21.15.135:6789/0 2024-02-06T22:17:57.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.120+0000 7fc9d181e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc9cc13fd30 0x7fc9cc146200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:57.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.120+0000 7fc9d181e640 10 monclient(hunting): picked mon.noname-a con 0x7fc9cc13fd30 addr v2:172.21.15.120:3300/0 2024-02-06T22:17:57.123 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.120+0000 7fc9d181e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9cc0a9f30 con 0x7fc9cc13f8d0 2024-02-06T22:17:57.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.120+0000 7fc9d181e640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:57.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.120+0000 7fc9d181e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9cc0b8c90 con 0x7fc9cc13fd30 2024-02-06T22:17:57.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.120+0000 7fc9d181e640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:57.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.120+0000 7fc9d181e640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9cc146740 con 0x7fc9cc0b88b0 2024-02-06T22:17:57.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.120+0000 7fc9d181e640 10 monclient(hunting): _renew_subs 2024-02-06T22:17:57.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.120+0000 7fc9d181e640 10 monclient(hunting): authenticate will time out at 3507.457607s 2024-02-06T22:17:57.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.120+0000 7fc9cb7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc9cc13fd30 0x7fc9cc146200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:57.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.120+0000 7fc9caffd640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc9cc0b88b0 0x7fc9cc0a99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:57.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9cb7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc9cc13fd30 0x7fc9cc146200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:41460/0 (socket says 172.21.15.120:41460) 2024-02-06T22:17:57.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9cb7fe640 1 -- 172.21.15.120:0/2260487397 learned_addr learned my addr 172.21.15.120:0/2260487397 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:17:57.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9cb7fe640 10 monclient(hunting): get_auth_request con 0x7fc9cc13fd30 auth_method 0 2024-02-06T22:17:57.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:57.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9cb7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:57.124 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:57.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9caffd640 10 monclient(hunting): get_auth_request con 0x7fc9cc0b88b0 auth_method 0 2024-02-06T22:17:57.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:57.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9caffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:57.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9caffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:57.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:57.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:57.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:57.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:57.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:57.125 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.121+0000 7fc9caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:57.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.123+0000 7fc9cb7fe640 10 monclient(hunting): handle_auth_done global_id 16482 payload 274 2024-02-06T22:17:57.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.123+0000 7fc9cb7fe640 1 -- 172.21.15.120:0/2260487397 >> v2:172.21.15.135:3300/0 conn(0x7fc9cc0b88b0 msgr2=0x7fc9cc0a99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:17:57.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.123+0000 7fc9cb7fe640 1 --2- 172.21.15.120:0/2260487397 >> v2:172.21.15.135:3300/0 conn(0x7fc9cc0b88b0 0x7fc9cc0a99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.123+0000 7fc9cb7fe640 1 -- 172.21.15.120:0/2260487397 >> v1:172.21.15.135:6789/0 conn(0x7fc9cc13f8d0 legacy=0x7fc9cc145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:57.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.123+0000 7fc9cb7fe640 10 monclient: _finish_hunting 0 2024-02-06T22:17:57.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.123+0000 7fc9cb7fe640 1 monclient: found mon.noname-a 2024-02-06T22:17:57.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.123+0000 7fc9cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:17:57.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.123+0000 7fc9cb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:17:57.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.123+0000 7fc9cb7fe640 1 -- 172.21.15.120:0/2260487397 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9cc0ab3b0 con 0x7fc9cc13fd30 2024-02-06T22:17:57.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.125+0000 7fc9cb7fe640 1 --2- 172.21.15.120:0/2260487397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9cc13fd30 0x7fc9cc146200 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc0b5ad0 tx=0x7fc9bc02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb0c8cf55dea8cd server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:17:57.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9c9ffb640 1 -- 172.21.15.120:0/2260487397 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc02cbb0 con 0x7fc9cc13fd30 2024-02-06T22:17:57.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:57.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9c9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:17:57.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9c9ffb640 10 monclient: dump: 2024-02-06T22:17:57.135 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:57.135 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:57.135 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.135 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.135 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:57.136 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:57.136 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:57.136 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:57.136 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:57.136 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:57.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9c9ffb640 10 monclient: _finish_auth 0 2024-02-06T22:17:57.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9c9ffb640 10 monclient: _check_auth_tickets 2024-02-06T22:17:57.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:17:57.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9c9ffb640 1 -- 172.21.15.120:0/2260487397 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc02cd10 con 0x7fc9cc13fd30 2024-02-06T22:17:57.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9d181e640 5 monclient: authenticate success, global_id 16482 2024-02-06T22:17:57.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9d181e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T22:17:57.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:17:57.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9d181e640 10 monclient: get_monmap_and_config success 2024-02-06T22:17:57.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9c9ffb640 1 -- 172.21.15.120:0/2260487397 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc034610 con 0x7fc9cc13fd30 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9d181e640 4 set_mon_vals no callback set 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9c9ffb640 10 monclient: dump: 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9d181e640 10 monclient: shutdown 2024-02-06T22:17:57.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9d181e640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:17:57.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9d181e640 1 -- 172.21.15.120:0/2260487397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9cc13fd30 msgr2=0x7fc9cc146200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:57.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9d181e640 1 --2- 172.21.15.120:0/2260487397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9cc13fd30 0x7fc9cc146200 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc0b5ad0 tx=0x7fc9bc02c010 comp rx=0 tx=0).stop 2024-02-06T22:17:57.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9d181e640 1 -- 172.21.15.120:0/2260487397 shutdown_connections 2024-02-06T22:17:57.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9d181e640 1 --2- 172.21.15.120:0/2260487397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9cc13fd30 0x7fc9cc146200 unknown :-1 s=CLOSED pgs=3341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9d181e640 1 --2- 172.21.15.120:0/2260487397 >> v2:172.21.15.135:3300/0 conn(0x7fc9cc0b88b0 0x7fc9cc0a99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9d181e640 1 -- 172.21.15.120:0/2260487397 >> 172.21.15.120:0/2260487397 conn(0x7fc9cc06d3b0 msgr2=0x7fc9cc0b7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:17:57.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9d181e640 1 -- 172.21.15.120:0/2260487397 shutdown_connections 2024-02-06T22:17:57.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.129+0000 7fc9d181e640 1 -- 172.21.15.120:0/2260487397 wait complete. 2024-02-06T22:17:57.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.130+0000 7fc9d181e640 10 monclient: build_initial_monmap 2024-02-06T22:17:57.138 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.130+0000 7fc9d181e640 10 monclient: monmap: 2024-02-06T22:17:57.138 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:17:57.138 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:17:57.138 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:17:57.138 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:17:57.139 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:17:57.139 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:57.139 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:17:57.139 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:17:57.139 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:17:57.139 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:57.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.130+0000 7fc9d181e640 1 librados: starting msgr at 2024-02-06T22:17:57.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.130+0000 7fc9d181e640 1 librados: starting objecter 2024-02-06T22:17:57.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.130+0000 7fc9d181e640 1 Processor -- start 2024-02-06T22:17:57.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 1 -- start start 2024-02-06T22:17:57.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 1 librados: setting wanted keys 2024-02-06T22:17:57.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 1 librados: calling monclient init 2024-02-06T22:17:57.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 10 monclient: init 2024-02-06T22:17:57.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 10 monclient: _reopen_session rank -1 2024-02-06T22:17:57.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T22:17:57.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9cc0b88b0 0x7fc9cc13b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:57.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 10 monclient(hunting): picked mon.noname-c con 0x7fc9cc0b88b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:17:57.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9cc13f8d0 0x7fc9cc1db390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:57.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 10 monclient(hunting): picked mon.noname-b con 0x7fc9cc13f8d0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:17:57.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9cc13fd30 0x7fc9cc1db8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:57.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 10 monclient(hunting): picked mon.noname-a con 0x7fc9cc13fd30 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:17:57.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:57.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc13beb0 con 0x7fc9cc13fd30 2024-02-06T22:17:57.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:57.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc1428a0 con 0x7fc9cc13f8d0 2024-02-06T22:17:57.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:57.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc142b80 con 0x7fc9cc0b88b0 2024-02-06T22:17:57.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 10 monclient(hunting): _renew_subs 2024-02-06T22:17:57.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.131+0000 7fc9d181e640 10 monclient(hunting): authenticate will time out at 3507.469278s 2024-02-06T22:17:57.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9cb7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9cc13fd30 0x7fc9cc1db8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:57.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9caffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9cc0b88b0 0x7fc9cc13b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:57.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9cb7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9cc13fd30 0x7fc9cc1db8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:41468/0 (socket says 172.21.15.120:41468) 2024-02-06T22:17:57.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9cb7fe640 1 -- 172.21.15.120:0/1372630628 learned_addr learned my addr 172.21.15.120:0/1372630628 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:17:57.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9cb7fe640 10 monclient(hunting): get_auth_request con 0x7fc9cc13fd30 auth_method 0 2024-02-06T22:17:57.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:57.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9cb7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:57.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:57.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9caffd640 10 monclient(hunting): get_auth_request con 0x7fc9cc0b88b0 auth_method 0 2024-02-06T22:17:57.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:57.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9caffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:57.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9caffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:57.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9ca7fc640 1 --2- 172.21.15.120:0/1372630628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9cc13f8d0 0x7fc9cc1db390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:57.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9ca7fc640 10 monclient(hunting): get_auth_request con 0x7fc9cc13f8d0 auth_method 0 2024-02-06T22:17:57.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:57.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9ca7fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:57.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:57.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.132+0000 7fc9cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:57.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.133+0000 7fc9cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:57.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.133+0000 7fc9cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:57.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.133+0000 7fc9caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:57.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.133+0000 7fc9caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:57.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.133+0000 7fc9caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:57.143 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.133+0000 7fc9ca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:57.143 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.133+0000 7fc9ca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:57.143 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.133+0000 7fc9ca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:57.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.135+0000 7fc9cb7fe640 10 monclient(hunting): handle_auth_done global_id 16485 payload 824 2024-02-06T22:17:57.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.135+0000 7fc9cb7fe640 1 -- 172.21.15.120:0/1372630628 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9cc0b88b0 msgr2=0x7fc9cc13b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:57.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.135+0000 7fc9cb7fe640 1 --2- 172.21.15.120:0/1372630628 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9cc0b88b0 0x7fc9cc13b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.135+0000 7fc9cb7fe640 1 -- 172.21.15.120:0/1372630628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9cc13f8d0 msgr2=0x7fc9cc1db390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:57.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.135+0000 7fc9cb7fe640 1 --2- 172.21.15.120:0/1372630628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9cc13f8d0 0x7fc9cc1db390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.135+0000 7fc9cb7fe640 10 monclient: _finish_hunting 0 2024-02-06T22:17:57.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.135+0000 7fc9cb7fe640 1 monclient: found mon.noname-a 2024-02-06T22:17:57.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.135+0000 7fc9cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:17:57.145 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.135+0000 7fc9cb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:17:57.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.135+0000 7fc9cb7fe640 1 -- 172.21.15.120:0/1372630628 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc009b30 con 0x7fc9cc13fd30 2024-02-06T22:17:57.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.137+0000 7fc9cb7fe640 1 --2- 172.21.15.120:0/1372630628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9cc13fd30 0x7fc9cc1db8d0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc036040 tx=0x7fc9bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:17:57.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.137+0000 7fc9b3fff640 10 client.?.objecter ms_handle_connect 0x7fc9cc13fd30 2024-02-06T22:17:57.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.137+0000 7fc9b3fff640 10 client.?.objecter resend_mon_ops 2024-02-06T22:17:57.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.139+0000 7fc9b3fff640 1 -- 172.21.15.120:0/1372630628 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc03e450 con 0x7fc9cc13fd30 2024-02-06T22:17:57.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.139+0000 7fc9b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:57.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.139+0000 7fc9b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:17:57.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.139+0000 7fc9b3fff640 10 monclient: dump: 2024-02-06T22:17:57.146 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:57.146 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:57.146 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.146 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.146 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:57.147 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:57.147 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:57.147 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:57.147 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:57.147 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:57.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9b3fff640 10 monclient: _finish_auth 0 2024-02-06T22:17:57.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9b3fff640 10 monclient: _check_auth_tickets 2024-02-06T22:17:57.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:17:57.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9d181e640 5 monclient: authenticate success, global_id 16485 2024-02-06T22:17:57.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9d181e640 10 monclient: _renew_subs 2024-02-06T22:17:57.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9d181e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:17:57.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9d181e640 1 -- 172.21.15.120:0/1372630628 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9cc141040 con 0x7fc9cc13fd30 2024-02-06T22:17:57.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9d181e640 10 client.16485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:17:57.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9d181e640 10 monclient: _renew_subs 2024-02-06T22:17:57.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9d181e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:17:57.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9d181e640 1 -- 172.21.15.120:0/1372630628 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9cc1dbe10 con 0x7fc9cc13fd30 2024-02-06T22:17:57.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9d181e640 1 librados: init done 2024-02-06T22:17:57.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9d181e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T22:17:57.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9d181e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:17:57.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9d181e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:17:57.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.140+0000 7fc9d181e640 1 -- 172.21.15.120:0/1372630628 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9cc04ef90 con 0x7fc9cc13fd30 2024-02-06T22:17:57.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.141+0000 7fc9b3fff640 1 -- 172.21.15.120:0/1372630628 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc03e5b0 con 0x7fc9cc13fd30 2024-02-06T22:17:57.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.141+0000 7fc9b3fff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:17:57.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.141+0000 7fc9b3fff640 1 -- 172.21.15.120:0/1372630628 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc03e920 con 0x7fc9cc13fd30 2024-02-06T22:17:57.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.141+0000 7fc9b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:57.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.141+0000 7fc9b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:17:57.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.141+0000 7fc9b3fff640 10 monclient: dump: 2024-02-06T22:17:57.148 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:57.149 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:57.149 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.149 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.149 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:57.149 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:57.149 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:57.149 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:57.149 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:57.149 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:57.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.141+0000 7fc9c8ff9640 4 set_mon_vals no callback set 2024-02-06T22:17:57.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.387+0000 7fc9b3fff640 1 -- 172.21.15.120:0/1372630628 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc9bc03ea80 con 0x7fc9cc13fd30 2024-02-06T22:17:57.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.388+0000 7fc9b3fff640 1 --2- 172.21.15.120:0/1372630628 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc9b806d1d0 0x7fc9b806f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:57.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.388+0000 7fc9caffd640 1 --2- 172.21.15.120:0/1372630628 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc9b806d1d0 0x7fc9b806f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:57.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.388+0000 7fc9caffd640 10 monclient: get_auth_request con 0x7fc9b806d1d0 auth_method 0 2024-02-06T22:17:57.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.388+0000 7fc9b3fff640 10 client.16485.objecter ms_handle_connect 0x7fc9b806d1d0 2024-02-06T22:17:57.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.388+0000 7fc9caffd640 1 --2- 172.21.15.120:0/1372630628 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc9b806d1d0 0x7fc9b806f690 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4009b40 tx=0x7fc9c4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:17:57.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.405+0000 7fc9b3fff640 1 -- 172.21.15.120:0/1372630628 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(1345..1345 src has 1047..1345) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fc9bc07bb60 con 0x7fc9cc13fd30 2024-02-06T22:17:57.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.405+0000 7fc9b3fff640 10 client.16485.objecter ms_dispatch 0x7fc9cc000b90 osd_map(1345..1345 src has 1047..1345) v4 2024-02-06T22:17:57.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.405+0000 7fc9b3fff640 3 client.16485.objecter handle_osd_map got epochs [1345,1345] > 0 2024-02-06T22:17:57.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.405+0000 7fc9b3fff640 3 client.16485.objecter handle_osd_map decoding full epoch 1345 2024-02-06T22:17:57.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.405+0000 7fc9b3fff640 20 client.16485.objecter dump_active .. 0 homeless 2024-02-06T22:17:57.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.405+0000 7fc9b3fff640 1 -- 172.21.15.120:0/1372630628 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc9bc077e00 con 0x7fc9cc13fd30 2024-02-06T22:17:57.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.405+0000 7fc9b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:17:57.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.405+0000 7fc9b3fff640 10 monclient: _finish_command 1 = system:0 2024-02-06T22:17:57.620 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.618+0000 7fc9d181e640 1 -- 172.21.15.120:0/1372630628 --> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc9cc142a10 con 0x7fc9b806d1d0 2024-02-06T22:17:57.626 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.624+0000 7fc9b3fff640 1 -- 172.21.15.120:0/1372630628 <== mgr.4100 v2:172.21.15.135:6800/315664144 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151689 (secure 0 0 0) 0x7fc9cc142a10 con 0x7fc9b806d1d0 2024-02-06T22:17:57.627 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T22:17:57.628 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2024-02-06T22:17:57.631 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 10 librados: watch_flush enter 2024-02-06T22:17:57.631 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 10 librados: watch_flush exit 2024-02-06T22:17:57.631 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 20 client.16485.objecter shutdown clearing up homeless session... 2024-02-06T22:17:57.631 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 10 client.16485.objecter successfully canceled tick 2024-02-06T22:17:57.631 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 1 -- 172.21.15.120:0/1372630628 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc9b806d1d0 msgr2=0x7fc9b806f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:57.631 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 1 --2- 172.21.15.120:0/1372630628 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc9b806d1d0 0x7fc9b806f690 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4009b40 tx=0x7fc9c4009190 comp rx=0 tx=0).stop 2024-02-06T22:17:57.631 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 10 monclient: shutdown 2024-02-06T22:17:57.631 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:17:57.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 1 -- 172.21.15.120:0/1372630628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9cc13fd30 msgr2=0x7fc9cc1db8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:57.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 1 --2- 172.21.15.120:0/1372630628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9cc13fd30 0x7fc9cc1db8d0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc036040 tx=0x7fc9bc004a40 comp rx=0 tx=0).stop 2024-02-06T22:17:57.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 1 -- 172.21.15.120:0/1372630628 shutdown_connections 2024-02-06T22:17:57.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 1 --2- 172.21.15.120:0/1372630628 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fc9b806d1d0 0x7fc9b806f690 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 1 --2- 172.21.15.120:0/1372630628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9cc13f8d0 0x7fc9cc1db390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 1 --2- 172.21.15.120:0/1372630628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9cc13fd30 0x7fc9cc1db8d0 unknown :-1 s=CLOSED pgs=3342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 1 --2- 172.21.15.120:0/1372630628 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9cc0b88b0 0x7fc9cc13b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 1 -- 172.21.15.120:0/1372630628 >> 172.21.15.120:0/1372630628 conn(0x7fc9cc06d3b0 msgr2=0x7fc9cc145f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:17:57.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 1 -- 172.21.15.120:0/1372630628 shutdown_connections 2024-02-06T22:17:57.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 1 -- 172.21.15.120:0/1372630628 wait complete. 2024-02-06T22:17:57.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.626+0000 7fc9d181e640 1 librados: shutdown 2024-02-06T22:17:57.639 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":2447,"stamp":"2024-02-06T22:17:56.396453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":582792,"kb_used_data":40416,"kb_used_omap":12,"kb_used_meta":542259,"kb_avail":754391928,"statfs":{"total":773094113280,"available":772497334272,"internally_reserved":0,"allocated":41385984,"data_stored":19661387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":555273776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":135,"apply_latency_ms":135,"commit_latency_ns":135000000,"apply_latency_ns":135000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.058711"},"pg_stats":[{"pgid":"12.1c","version":"0'0","reported_seq":2582,"reported_epoch":1340,"state":"active+clean","last_fresh":"2024-02-06T22:17:51.319476+0000","last_change":"2024-02-06T21:42:50.083729+0000","last_active":"2024-02-06T22:17:51.319476+0000","last_peered":"2024-02-06T22:17:51.319476+0000","last_clean":"2024-02-06T22:17:51.319476+0000","last_became_active":"2024-02-06T21:42:50.078423+0000","last_became_peered":"2024-02-06T21:42:50.078423+0000","last_unstale":"2024-02-06T22:17:51.319476+0000","last_undegraded":"2024-02-06T22:17:51.319476+0000","last_fullsized":"2024-02-06T22:17:51.319476+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:44:16.757427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":2580,"reported_epoch":1343,"state":"active+clean","last_fresh":"2024-02-06T22:17:54.405293+0000","last_change":"2024-02-06T21:43:19.779411+0000","last_active":"2024-02-06T22:17:54.405293+0000","last_peered":"2024-02-06T22:17:54.405293+0000","last_clean":"2024-02-06T22:17:54.405293+0000","last_became_active":"2024-02-06T21:43:19.773686+0000","last_became_peered":"2024-02-06T21:43:19.773686+0000","last_unstale":"2024-02-06T22:17:54.405293+0000","last_undegraded":"2024-02-06T22:17:54.405293+0000","last_fullsized":"2024-02-06T22:17:54.405293+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:54:24.026659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":2585,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.382909+0000","last_change":"2024-02-06T21:42:53.308445+0000","last_active":"2024-02-06T22:17:53.382909+0000","last_peered":"2024-02-06T22:17:53.382909+0000","last_clean":"2024-02-06T22:17:53.382909+0000","last_became_active":"2024-02-06T21:42:53.291194+0000","last_became_peered":"2024-02-06T21:42:53.291194+0000","last_unstale":"2024-02-06T22:17:53.382909+0000","last_undegraded":"2024-02-06T22:17:53.382909+0000","last_fullsized":"2024-02-06T22:17:53.382909+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T00:08:35.742342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":2575,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.511147+0000","last_change":"2024-02-06T21:42:45.080259+0000","last_active":"2024-02-06T22:17:53.511147+0000","last_peered":"2024-02-06T22:17:53.511147+0000","last_clean":"2024-02-06T22:17:53.511147+0000","last_became_active":"2024-02-06T21:42:45.065933+0000","last_became_peered":"2024-02-06T21:42:45.065933+0000","last_unstale":"2024-02-06T22:17:53.511147+0000","last_undegraded":"2024-02-06T22:17:53.511147+0000","last_fullsized":"2024-02-06T22:17:53.511147+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:44:14.566236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":2566,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.510473+0000","last_change":"2024-02-06T21:42:58.554516+0000","last_active":"2024-02-06T22:17:50.510473+0000","last_peered":"2024-02-06T22:17:50.510473+0000","last_clean":"2024-02-06T22:17:50.510473+0000","last_became_active":"2024-02-06T21:42:58.548361+0000","last_became_peered":"2024-02-06T21:42:58.548361+0000","last_unstale":"2024-02-06T22:17:50.510473+0000","last_undegraded":"2024-02-06T22:17:50.510473+0000","last_fullsized":"2024-02-06T22:17:50.510473+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T07:18:58.848738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"124'7","reported_seq":2587,"reported_epoch":1341,"state":"active+clean","last_fresh":"2024-02-06T22:17:52.364003+0000","last_change":"2024-02-06T21:43:15.527724+0000","last_active":"2024-02-06T22:17:52.364003+0000","last_peered":"2024-02-06T22:17:52.364003+0000","last_clean":"2024-02-06T22:17:52.364003+0000","last_became_active":"2024-02-06T21:43:15.497081+0000","last_became_peered":"2024-02-06T21:43:15.497081+0000","last_unstale":"2024-02-06T22:17:52.364003+0000","last_undegraded":"2024-02-06T22:17:52.364003+0000","last_fullsized":"2024-02-06T22:17:52.364003+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:53:37.062280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":2586,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.365646+0000","last_change":"2024-02-06T21:43:13.230133+0000","last_active":"2024-02-06T22:17:53.365646+0000","last_peered":"2024-02-06T22:17:53.365646+0000","last_clean":"2024-02-06T22:17:53.365646+0000","last_became_active":"2024-02-06T21:43:13.108638+0000","last_became_peered":"2024-02-06T21:43:13.108638+0000","last_unstale":"2024-02-06T22:17:53.365646+0000","last_undegraded":"2024-02-06T22:17:53.365646+0000","last_fullsized":"2024-02-06T22:17:53.365646+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:24:54.727933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":2593,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.377205+0000","last_change":"2024-02-06T21:43:06.276185+0000","last_active":"2024-02-06T22:17:53.377205+0000","last_peered":"2024-02-06T22:17:53.377205+0000","last_clean":"2024-02-06T22:17:53.377205+0000","last_became_active":"2024-02-06T21:43:06.270947+0000","last_became_peered":"2024-02-06T21:43:06.270947+0000","last_unstale":"2024-02-06T22:17:53.377205+0000","last_undegraded":"2024-02-06T22:17:53.377205+0000","last_fullsized":"2024-02-06T22:17:53.377205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:27:26.086535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":2586,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.379590+0000","last_change":"2024-02-06T21:42:53.563677+0000","last_active":"2024-02-06T22:17:53.379590+0000","last_peered":"2024-02-06T22:17:53.379590+0000","last_clean":"2024-02-06T22:17:53.379590+0000","last_became_active":"2024-02-06T21:42:53.535059+0000","last_became_peered":"2024-02-06T21:42:53.535059+0000","last_unstale":"2024-02-06T22:17:53.379590+0000","last_undegraded":"2024-02-06T22:17:53.379590+0000","last_fullsized":"2024-02-06T22:17:53.379590+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T06:13:12.610796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":2568,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.521205+0000","last_change":"2024-02-06T21:42:46.177846+0000","last_active":"2024-02-06T22:17:50.521205+0000","last_peered":"2024-02-06T22:17:50.521205+0000","last_clean":"2024-02-06T22:17:50.521205+0000","last_became_active":"2024-02-06T21:42:46.168640+0000","last_became_peered":"2024-02-06T21:42:46.168640+0000","last_unstale":"2024-02-06T22:17:50.521205+0000","last_undegraded":"2024-02-06T22:17:50.521205+0000","last_fullsized":"2024-02-06T22:17:50.521205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:37:19.536133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":2572,"reported_epoch":1343,"state":"active+clean","last_fresh":"2024-02-06T22:17:54.417919+0000","last_change":"2024-02-06T21:43:27.891680+0000","last_active":"2024-02-06T22:17:54.417919+0000","last_peered":"2024-02-06T22:17:54.417919+0000","last_clean":"2024-02-06T22:17:54.417919+0000","last_became_active":"2024-02-06T21:43:27.883526+0000","last_became_peered":"2024-02-06T21:43:27.883526+0000","last_unstale":"2024-02-06T22:17:54.417919+0000","last_undegraded":"2024-02-06T22:17:54.417919+0000","last_fullsized":"2024-02-06T22:17:54.417919+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:21:41.402059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":2570,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.530267+0000","last_change":"2024-02-06T21:42:45.882433+0000","last_active":"2024-02-06T22:17:50.530267+0000","last_peered":"2024-02-06T22:17:50.530267+0000","last_clean":"2024-02-06T22:17:50.530267+0000","last_became_active":"2024-02-06T21:42:45.868987+0000","last_became_peered":"2024-02-06T21:42:45.868987+0000","last_unstale":"2024-02-06T22:17:50.530267+0000","last_undegraded":"2024-02-06T22:17:50.530267+0000","last_fullsized":"2024-02-06T22:17:50.530267+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T00:26:41.132150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":2572,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.498068+0000","last_change":"2024-02-06T21:42:59.955119+0000","last_active":"2024-02-06T22:17:53.498068+0000","last_peered":"2024-02-06T22:17:53.498068+0000","last_clean":"2024-02-06T22:17:53.498068+0000","last_became_active":"2024-02-06T21:42:59.949344+0000","last_became_peered":"2024-02-06T21:42:59.949344+0000","last_unstale":"2024-02-06T22:17:53.498068+0000","last_undegraded":"2024-02-06T22:17:53.498068+0000","last_fullsized":"2024-02-06T22:17:53.498068+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T06:07:53.514679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'76","reported_seq":2694,"reported_epoch":1341,"state":"active+clean","last_fresh":"2024-02-06T22:17:52.355749+0000","last_change":"2024-02-06T21:41:31.339576+0000","last_active":"2024-02-06T22:17:52.355749+0000","last_peered":"2024-02-06T22:17:52.355749+0000","last_clean":"2024-02-06T22:17:52.355749+0000","last_became_active":"2024-02-06T21:41:31.305766+0000","last_became_peered":"2024-02-06T21:41:31.305766+0000","last_unstale":"2024-02-06T22:17:52.355749+0000","last_undegraded":"2024-02-06T22:17:52.355749+0000","last_fullsized":"2024-02-06T22:17:52.355749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:20.409569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:20.409569+0000","last_clean_scrub_stamp":"2024-02-06T21:41:20.409569+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:24:57.767580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2586,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.504352+0000","last_change":"2024-02-06T21:41:36.168152+0000","last_active":"2024-02-06T22:17:53.504352+0000","last_peered":"2024-02-06T22:17:53.504352+0000","last_clean":"2024-02-06T22:17:53.504352+0000","last_became_active":"2024-02-06T21:41:36.162988+0000","last_became_peered":"2024-02-06T21:41:36.162988+0000","last_unstale":"2024-02-06T22:17:53.504352+0000","last_undegraded":"2024-02-06T22:17:53.504352+0000","last_fullsized":"2024-02-06T22:17:53.504352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T00:47:41.526670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":2569,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.526919+0000","last_change":"2024-02-06T21:42:56.031740+0000","last_active":"2024-02-06T22:17:50.526919+0000","last_peered":"2024-02-06T22:17:50.526919+0000","last_clean":"2024-02-06T22:17:50.526919+0000","last_became_active":"2024-02-06T21:42:56.023045+0000","last_became_peered":"2024-02-06T21:42:56.023045+0000","last_unstale":"2024-02-06T22:17:50.526919+0000","last_undegraded":"2024-02-06T22:17:50.526919+0000","last_fullsized":"2024-02-06T22:17:50.526919+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:19:09.477603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2587,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.467482+0000","last_change":"2024-02-06T21:41:35.688242+0000","last_active":"2024-02-06T22:17:53.467482+0000","last_peered":"2024-02-06T22:17:53.467482+0000","last_clean":"2024-02-06T22:17:53.467482+0000","last_became_active":"2024-02-06T21:41:35.642164+0000","last_became_peered":"2024-02-06T21:41:35.642164+0000","last_unstale":"2024-02-06T22:17:53.467482+0000","last_undegraded":"2024-02-06T22:17:53.467482+0000","last_fullsized":"2024-02-06T22:17:53.467482+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:50:03.838525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":2569,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.512725+0000","last_change":"2024-02-06T21:43:05.426147+0000","last_active":"2024-02-06T22:17:53.512725+0000","last_peered":"2024-02-06T22:17:53.512725+0000","last_clean":"2024-02-06T22:17:53.512725+0000","last_became_active":"2024-02-06T21:43:05.414230+0000","last_became_peered":"2024-02-06T21:43:05.414230+0000","last_unstale":"2024-02-06T22:17:53.512725+0000","last_undegraded":"2024-02-06T22:17:53.512725+0000","last_fullsized":"2024-02-06T22:17:53.512725+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:04:00.128718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2581,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.508463+0000","last_change":"2024-02-06T21:41:35.636555+0000","last_active":"2024-02-06T22:17:50.508463+0000","last_peered":"2024-02-06T22:17:50.508463+0000","last_clean":"2024-02-06T22:17:50.508463+0000","last_became_active":"2024-02-06T21:41:35.603135+0000","last_became_peered":"2024-02-06T21:41:35.603135+0000","last_unstale":"2024-02-06T22:17:50.508463+0000","last_undegraded":"2024-02-06T22:17:50.508463+0000","last_fullsized":"2024-02-06T22:17:50.508463+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T03:13:14.699865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":2570,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.522055+0000","last_change":"2024-02-06T21:42:49.418971+0000","last_active":"2024-02-06T22:17:50.522055+0000","last_peered":"2024-02-06T22:17:50.522055+0000","last_clean":"2024-02-06T22:17:50.522055+0000","last_became_active":"2024-02-06T21:42:49.409862+0000","last_became_peered":"2024-02-06T21:42:49.409862+0000","last_unstale":"2024-02-06T22:17:50.522055+0000","last_undegraded":"2024-02-06T22:17:50.522055+0000","last_fullsized":"2024-02-06T22:17:50.522055+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:08:26.530828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2587,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.487524+0000","last_change":"2024-02-06T21:41:35.892547+0000","last_active":"2024-02-06T22:17:53.487524+0000","last_peered":"2024-02-06T22:17:53.487524+0000","last_clean":"2024-02-06T22:17:53.487524+0000","last_became_active":"2024-02-06T21:41:35.877657+0000","last_became_peered":"2024-02-06T21:41:35.877657+0000","last_unstale":"2024-02-06T22:17:53.487524+0000","last_undegraded":"2024-02-06T22:17:53.487524+0000","last_fullsized":"2024-02-06T22:17:53.487524+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:23:28.851375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":2593,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.383234+0000","last_change":"2024-02-06T21:42:56.499357+0000","last_active":"2024-02-06T22:17:53.383234+0000","last_peered":"2024-02-06T22:17:53.383234+0000","last_clean":"2024-02-06T22:17:53.383234+0000","last_became_active":"2024-02-06T21:42:56.473273+0000","last_became_peered":"2024-02-06T21:42:56.473273+0000","last_unstale":"2024-02-06T22:17:53.383234+0000","last_undegraded":"2024-02-06T22:17:53.383234+0000","last_fullsized":"2024-02-06T22:17:53.383234+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:45:45.452902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2594,"reported_epoch":1340,"state":"active+clean","last_fresh":"2024-02-06T22:17:51.321473+0000","last_change":"2024-02-06T21:41:35.918723+0000","last_active":"2024-02-06T22:17:51.321473+0000","last_peered":"2024-02-06T22:17:51.321473+0000","last_clean":"2024-02-06T22:17:51.321473+0000","last_became_active":"2024-02-06T21:41:35.891748+0000","last_became_peered":"2024-02-06T21:41:35.891748+0000","last_unstale":"2024-02-06T22:17:51.321473+0000","last_undegraded":"2024-02-06T22:17:51.321473+0000","last_fullsized":"2024-02-06T22:17:51.321473+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T03:59:11.081636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":2570,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.510479+0000","last_change":"2024-02-06T21:42:49.677946+0000","last_active":"2024-02-06T22:17:50.510479+0000","last_peered":"2024-02-06T22:17:50.510479+0000","last_clean":"2024-02-06T22:17:50.510479+0000","last_became_active":"2024-02-06T21:42:49.671861+0000","last_became_peered":"2024-02-06T21:42:49.671861+0000","last_unstale":"2024-02-06T22:17:50.510479+0000","last_undegraded":"2024-02-06T22:17:50.510479+0000","last_fullsized":"2024-02-06T22:17:50.510479+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:43:48.362335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2586,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.499796+0000","last_change":"2024-02-06T21:41:35.766413+0000","last_active":"2024-02-06T22:17:53.499796+0000","last_peered":"2024-02-06T22:17:53.499796+0000","last_clean":"2024-02-06T22:17:53.499796+0000","last_became_active":"2024-02-06T21:41:35.761324+0000","last_became_peered":"2024-02-06T21:41:35.761324+0000","last_unstale":"2024-02-06T22:17:53.499796+0000","last_undegraded":"2024-02-06T22:17:53.499796+0000","last_fullsized":"2024-02-06T22:17:53.499796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:28:07.018583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":2571,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.512477+0000","last_change":"2024-02-06T21:42:44.700963+0000","last_active":"2024-02-06T22:17:50.512477+0000","last_peered":"2024-02-06T22:17:50.512477+0000","last_clean":"2024-02-06T22:17:50.512477+0000","last_became_active":"2024-02-06T21:42:44.691217+0000","last_became_peered":"2024-02-06T21:42:44.691217+0000","last_unstale":"2024-02-06T22:17:50.512477+0000","last_undegraded":"2024-02-06T22:17:50.512477+0000","last_fullsized":"2024-02-06T22:17:50.512477+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:15:55.323353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2609,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.390113+0000","last_change":"2024-02-06T21:41:36.193098+0000","last_active":"2024-02-06T22:17:53.390113+0000","last_peered":"2024-02-06T22:17:53.390113+0000","last_clean":"2024-02-06T22:17:53.390113+0000","last_became_active":"2024-02-06T21:41:36.160648+0000","last_became_peered":"2024-02-06T21:41:36.160648+0000","last_unstale":"2024-02-06T22:17:53.390113+0000","last_undegraded":"2024-02-06T22:17:53.390113+0000","last_fullsized":"2024-02-06T22:17:53.390113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T09:01:03.161738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":2582,"reported_epoch":1343,"state":"active+clean","last_fresh":"2024-02-06T22:17:54.409582+0000","last_change":"2024-02-06T21:42:52.618077+0000","last_active":"2024-02-06T22:17:54.409582+0000","last_peered":"2024-02-06T22:17:54.409582+0000","last_clean":"2024-02-06T22:17:54.409582+0000","last_became_active":"2024-02-06T21:42:52.611794+0000","last_became_peered":"2024-02-06T21:42:52.611794+0000","last_unstale":"2024-02-06T22:17:54.409582+0000","last_undegraded":"2024-02-06T22:17:54.409582+0000","last_fullsized":"2024-02-06T22:17:54.409582+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:25:35.938955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2583,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.514531+0000","last_change":"2024-02-06T21:41:36.144443+0000","last_active":"2024-02-06T22:17:53.514531+0000","last_peered":"2024-02-06T22:17:53.514531+0000","last_clean":"2024-02-06T22:17:53.514531+0000","last_became_active":"2024-02-06T21:41:36.139312+0000","last_became_peered":"2024-02-06T21:41:36.139312+0000","last_unstale":"2024-02-06T22:17:53.514531+0000","last_undegraded":"2024-02-06T22:17:53.514531+0000","last_fullsized":"2024-02-06T22:17:53.514531+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:47:25.348858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":2567,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.517568+0000","last_change":"2024-02-06T21:42:49.714094+0000","last_active":"2024-02-06T22:17:50.517568+0000","last_peered":"2024-02-06T22:17:50.517568+0000","last_clean":"2024-02-06T22:17:50.517568+0000","last_became_active":"2024-02-06T21:42:49.708820+0000","last_became_peered":"2024-02-06T21:42:49.708820+0000","last_unstale":"2024-02-06T22:17:50.517568+0000","last_undegraded":"2024-02-06T22:17:50.517568+0000","last_fullsized":"2024-02-06T22:17:50.517568+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:55:54.917799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":2584,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.362656+0000","last_change":"2024-02-06T21:43:13.244098+0000","last_active":"2024-02-06T22:17:53.362656+0000","last_peered":"2024-02-06T22:17:53.362656+0000","last_clean":"2024-02-06T22:17:53.362656+0000","last_became_active":"2024-02-06T21:43:13.123786+0000","last_became_peered":"2024-02-06T21:43:13.123786+0000","last_unstale":"2024-02-06T22:17:53.362656+0000","last_undegraded":"2024-02-06T22:17:53.362656+0000","last_fullsized":"2024-02-06T22:17:53.362656+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T02:53:35.916469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":2565,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.506337+0000","last_change":"2024-02-06T21:42:56.872139+0000","last_active":"2024-02-06T22:17:50.506337+0000","last_peered":"2024-02-06T22:17:50.506337+0000","last_clean":"2024-02-06T22:17:50.506337+0000","last_became_active":"2024-02-06T21:42:56.865960+0000","last_became_peered":"2024-02-06T21:42:56.865960+0000","last_unstale":"2024-02-06T22:17:50.506337+0000","last_undegraded":"2024-02-06T22:17:50.506337+0000","last_fullsized":"2024-02-06T22:17:50.506337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:13:16.665309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":2569,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.514243+0000","last_change":"2024-02-06T21:42:45.712485+0000","last_active":"2024-02-06T22:17:50.514243+0000","last_peered":"2024-02-06T22:17:50.514243+0000","last_clean":"2024-02-06T22:17:50.514243+0000","last_became_active":"2024-02-06T21:42:45.706432+0000","last_became_peered":"2024-02-06T21:42:45.706432+0000","last_unstale":"2024-02-06T22:17:50.514243+0000","last_undegraded":"2024-02-06T22:17:50.514243+0000","last_fullsized":"2024-02-06T22:17:50.514243+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:46:59.377729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":2569,"reported_epoch":1341,"state":"active+clean","last_fresh":"2024-02-06T22:17:52.368275+0000","last_change":"2024-02-06T21:43:08.879863+0000","last_active":"2024-02-06T22:17:52.368275+0000","last_peered":"2024-02-06T22:17:52.368275+0000","last_clean":"2024-02-06T22:17:52.368275+0000","last_became_active":"2024-02-06T21:43:08.871254+0000","last_became_peered":"2024-02-06T21:43:08.871254+0000","last_unstale":"2024-02-06T22:17:52.368275+0000","last_undegraded":"2024-02-06T22:17:52.368275+0000","last_fullsized":"2024-02-06T22:17:52.368275+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:02:54.631582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":2582,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.384957+0000","last_change":"2024-02-06T21:42:55.060952+0000","last_active":"2024-02-06T22:17:53.384957+0000","last_peered":"2024-02-06T22:17:53.384957+0000","last_clean":"2024-02-06T22:17:53.384957+0000","last_became_active":"2024-02-06T21:42:55.034062+0000","last_became_peered":"2024-02-06T21:42:55.034062+0000","last_unstale":"2024-02-06T22:17:53.384957+0000","last_undegraded":"2024-02-06T22:17:53.384957+0000","last_fullsized":"2024-02-06T22:17:53.384957+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T07:52:53.348265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":2572,"reported_epoch":1341,"state":"active+clean","last_fresh":"2024-02-06T22:17:52.376138+0000","last_change":"2024-02-06T21:43:15.867032+0000","last_active":"2024-02-06T22:17:52.376138+0000","last_peered":"2024-02-06T22:17:52.376138+0000","last_clean":"2024-02-06T22:17:52.376138+0000","last_became_active":"2024-02-06T21:43:15.861140+0000","last_became_peered":"2024-02-06T21:43:15.861140+0000","last_unstale":"2024-02-06T22:17:52.376138+0000","last_undegraded":"2024-02-06T22:17:52.376138+0000","last_fullsized":"2024-02-06T22:17:52.376138+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:00:40.646612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":2568,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.529638+0000","last_change":"2024-02-06T21:42:55.175403+0000","last_active":"2024-02-06T22:17:50.529638+0000","last_peered":"2024-02-06T22:17:50.529638+0000","last_clean":"2024-02-06T22:17:50.529638+0000","last_became_active":"2024-02-06T21:42:55.168677+0000","last_became_peered":"2024-02-06T21:42:55.168677+0000","last_unstale":"2024-02-06T22:17:50.529638+0000","last_undegraded":"2024-02-06T22:17:50.529638+0000","last_fullsized":"2024-02-06T22:17:50.529638+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T07:31:47.782588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":2583,"reported_epoch":1340,"state":"active+clean","last_fresh":"2024-02-06T22:17:51.323256+0000","last_change":"2024-02-06T21:42:52.000053+0000","last_active":"2024-02-06T22:17:51.323256+0000","last_peered":"2024-02-06T22:17:51.323256+0000","last_clean":"2024-02-06T22:17:51.323256+0000","last_became_active":"2024-02-06T21:42:51.993812+0000","last_became_peered":"2024-02-06T21:42:51.993812+0000","last_unstale":"2024-02-06T22:17:51.323256+0000","last_undegraded":"2024-02-06T22:17:51.323256+0000","last_fullsized":"2024-02-06T22:17:51.323256+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:44:21.306346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":2572,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.382261+0000","last_change":"2024-02-06T21:44:01.005089+0000","last_active":"2024-02-06T22:17:53.382261+0000","last_peered":"2024-02-06T22:17:53.382261+0000","last_clean":"2024-02-06T22:17:53.382261+0000","last_became_active":"2024-02-06T21:44:00.992836+0000","last_became_peered":"2024-02-06T21:44:00.992836+0000","last_unstale":"2024-02-06T22:17:53.382261+0000","last_undegraded":"2024-02-06T22:17:53.382261+0000","last_fullsized":"2024-02-06T22:17:53.382261+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:34:43.433551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":2580,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.359066+0000","last_change":"2024-02-06T21:43:14.066712+0000","last_active":"2024-02-06T22:17:53.359066+0000","last_peered":"2024-02-06T22:17:53.359066+0000","last_clean":"2024-02-06T22:17:53.359066+0000","last_became_active":"2024-02-06T21:43:14.057258+0000","last_became_peered":"2024-02-06T21:43:14.057258+0000","last_unstale":"2024-02-06T22:17:53.359066+0000","last_undegraded":"2024-02-06T22:17:53.359066+0000","last_fullsized":"2024-02-06T22:17:53.359066+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T21:49:47.557758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":2582,"reported_epoch":1343,"state":"active+clean","last_fresh":"2024-02-06T22:17:54.416037+0000","last_change":"2024-02-06T21:42:49.763089+0000","last_active":"2024-02-06T22:17:54.416037+0000","last_peered":"2024-02-06T22:17:54.416037+0000","last_clean":"2024-02-06T22:17:54.416037+0000","last_became_active":"2024-02-06T21:42:49.754000+0000","last_became_peered":"2024-02-06T21:42:49.754000+0000","last_unstale":"2024-02-06T22:17:54.416037+0000","last_undegraded":"2024-02-06T22:17:54.416037+0000","last_fullsized":"2024-02-06T22:17:54.416037+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T06:23:56.755422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":18,"seq":77309411767,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72632,"kb_used_data":5352,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299208,"statfs":{"total":96636764160,"available":96562388992,"internally_reserved":0,"allocated":5480448,"data_stored":2758305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:41 2024","interfaces":[{"interface":"back","average":{"1min":15.664999999999999,"5min":29.364000000000001,"15min":24.5},"min":{"1min":5.9740000000000002,"5min":4.96,"15min":4.5209999999999999},"max":{"1min":76.638000000000005,"5min":223.274,"15min":549.32299999999998},"last":9.5389999999999997},{"interface":"front","average":{"1min":20.262,"5min":35.253999999999998,"15min":29.021999999999998},"min":{"1min":10.795999999999999,"5min":7.9969999999999999,"15min":5.4900000000000002},"max":{"1min":78.369,"5min":224.31700000000001,"15min":368.31200000000001},"last":16.876000000000001}]},{"osd":2,"last update":"Tue Feb 6 22:17:09 2024","interfaces":[{"interface":"back","average":{"1min":23.908999999999999,"5min":30.920999999999999,"15min":26.542999999999999},"min":{"1min":11.26,"5min":5.1520000000000001,"15min":5.1520000000000001},"max":{"1min":88.763000000000005,"5min":205.28800000000001,"15min":387.75299999999999},"last":11.986000000000001},{"interface":"front","average":{"1min":12.706,"5min":29.155000000000001,"15min":31.622},"min":{"1min":5.3460000000000001,"5min":5.3460000000000001,"15min":5.3460000000000001},"max":{"1min":43.438000000000002,"5min":220.31899999999999,"15min":442.25},"last":13.981999999999999}]},{"osd":3,"last update":"Tue Feb 6 22:16:51 2024","interfaces":[{"interface":"back","average":{"1min":12.071,"5min":24.050999999999998,"15min":22.372},"min":{"1min":5.3899999999999997,"5min":5.3899999999999997,"15min":5.3899999999999997},"max":{"1min":27.318000000000001,"5min":359.19299999999998,"15min":359.19299999999998},"last":10.957000000000001},{"interface":"front","average":{"1min":24.745999999999999,"5min":33.384999999999998,"15min":31.391999999999999},"min":{"1min":11.438000000000001,"5min":8.5500000000000007,"15min":7.8010000000000002},"max":{"1min":100.857,"5min":266.40600000000001,"15min":462.86200000000002},"last":18.096}]},{"osd":4,"last update":"Tue Feb 6 22:17:24 2024","interfaces":[{"interface":"back","average":{"1min":25.393999999999998,"5min":39.593000000000004,"15min":29.221},"min":{"1min":12.670999999999999,"5min":9.0559999999999992,"15min":5.9539999999999997},"max":{"1min":85.082999999999998,"5min":535.22500000000002,"15min":535.22500000000002},"last":20.530000000000001},{"interface":"front","average":{"1min":12.597,"5min":28.677,"15min":22.689},"min":{"1min":7.5220000000000002,"5min":6.3010000000000002,"15min":5.4649999999999999},"max":{"1min":49.466000000000001,"5min":473.69799999999998,"15min":473.69799999999998},"last":8.3940000000000001}]},{"osd":5,"last update":"Tue Feb 6 22:17:41 2024","interfaces":[{"interface":"back","average":{"1min":27.221,"5min":36.164999999999999,"15min":32.499000000000002},"min":{"1min":14.074999999999999,"5min":8.0459999999999994,"15min":6.4550000000000001},"max":{"1min":90.605999999999995,"5min":260.74400000000003,"15min":454.49099999999999},"last":21.324000000000002},{"interface":"front","average":{"1min":17.254000000000001,"5min":25.736999999999998,"15min":26.802},"min":{"1min":8.1069999999999993,"5min":6.8010000000000002,"15min":6.109},"max":{"1min":70.180000000000007,"5min":147.011,"15min":539.04700000000003},"last":14.577}]},{"osd":6,"last update":"Tue Feb 6 22:17:24 2024","interfaces":[{"interface":"back","average":{"1min":27.564,"5min":40.116999999999997,"15min":37.177},"min":{"1min":13.347,"5min":12.289,"15min":10.353999999999999},"max":{"1min":89.951999999999998,"5min":268.73399999999998,"15min":482.846},"last":22.105},{"interface":"front","average":{"1min":16.326000000000001,"5min":27.145,"15min":24.978000000000002},"min":{"1min":8.7759999999999998,"5min":6.9349999999999996,"15min":6.9349999999999996},"max":{"1min":51.412999999999997,"5min":186.95500000000001,"15min":322.64999999999998},"last":13.06}]},{"osd":7,"last update":"Tue Feb 6 22:17:41 2024","interfaces":[{"interface":"back","average":{"1min":15.381,"5min":27.943000000000001,"15min":30.135000000000002},"min":{"1min":6.1669999999999998,"5min":6.1669999999999998,"15min":6.1669999999999998},"max":{"1min":75.899000000000001,"5min":249.69200000000001,"15min":452.22399999999999},"last":6.3639999999999999},{"interface":"front","average":{"1min":31.131,"5min":42.412999999999997,"15min":35.392000000000003},"min":{"1min":13.923999999999999,"5min":10.964,"15min":7.7030000000000003},"max":{"1min":99.406000000000006,"5min":292.94600000000003,"15min":563.26900000000001},"last":19.073}]}]},{"osd":6,"up_from":17,"seq":73014444472,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72248,"kb_used_data":4972,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299592,"statfs":{"total":96636764160,"available":96562782208,"internally_reserved":0,"allocated":5091328,"data_stored":2367949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":14.135,"5min":21.481000000000002,"15min":17.268000000000001},"min":{"1min":4.3339999999999996,"5min":4.3230000000000004,"15min":4.3230000000000004},"max":{"1min":59.670999999999999,"5min":188.761,"15min":218.85400000000001},"last":13.481999999999999},{"interface":"front","average":{"1min":11.811,"5min":24.353000000000002,"15min":17.314},"min":{"1min":5.1559999999999997,"5min":5.1559999999999997,"15min":4.8449999999999998},"max":{"1min":38.082999999999998,"5min":193.68000000000001,"15min":193.68000000000001},"last":7.5709999999999997}]},{"osd":1,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":15.811999999999999,"5min":23.908999999999999,"15min":21.940000000000001},"min":{"1min":6.431,"5min":6.1459999999999999,"15min":5.109},"max":{"1min":54.000999999999998,"5min":180.447,"15min":204.64500000000001},"last":12.768000000000001},{"interface":"front","average":{"1min":17.199000000000002,"5min":23.283000000000001,"15min":24.012},"min":{"1min":6.7949999999999999,"5min":6.7949999999999999,"15min":6.1070000000000002},"max":{"1min":60.319000000000003,"5min":168.404,"15min":460.71600000000001},"last":8.9610000000000003}]},{"osd":2,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":19.161000000000001,"5min":36.042000000000002,"15min":26.562999999999999},"min":{"1min":8.0570000000000004,"5min":6.9359999999999999,"15min":5.5330000000000004},"max":{"1min":72.200000000000003,"5min":794.04200000000003,"15min":794.04200000000003},"last":11.016},{"interface":"front","average":{"1min":19.587,"5min":43.002000000000002,"15min":27.474},"min":{"1min":11.856,"5min":6.2999999999999998,"15min":5.601},"max":{"1min":62.237000000000002,"5min":795.17499999999995,"15min":795.17499999999995},"last":15.772}]},{"osd":3,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":19.741,"5min":32.057000000000002,"15min":24.117000000000001},"min":{"1min":10.901999999999999,"5min":7.718,"15min":5.9240000000000004},"max":{"1min":72.951999999999998,"5min":317.899,"15min":317.899},"last":11.653},{"interface":"front","average":{"1min":20.227,"5min":29.085999999999999,"15min":21.300999999999998},"min":{"1min":10.553000000000001,"5min":6.468,"15min":6.0380000000000003},"max":{"1min":65.974000000000004,"5min":405.108,"15min":405.108},"last":16.552}]},{"osd":4,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":10.769,"5min":23.568999999999999,"15min":22.443999999999999},"min":{"1min":5.8940000000000001,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":30.241,"5min":231.55699999999999,"15min":243.667},"last":6.4859999999999998},{"interface":"front","average":{"1min":14.613,"5min":30.960000000000001,"15min":27.713000000000001},"min":{"1min":8.641,"5min":7.7160000000000002,"15min":7.3399999999999999},"max":{"1min":39.484999999999999,"5min":284.37,"15min":339.34399999999999},"last":9.9689999999999994}]},{"osd":5,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":20.939,"5min":24.574999999999999,"15min":22.995000000000001},"min":{"1min":11.087999999999999,"5min":9.2929999999999993,"15min":7.5990000000000002},"max":{"1min":67.230000000000004,"5min":157.095,"15min":240.20500000000001},"last":14.972},{"interface":"front","average":{"1min":23.254999999999999,"5min":31.707999999999998,"15min":27.795000000000002},"min":{"1min":14.375,"5min":13.481,"15min":9.9260000000000002},"max":{"1min":69.085999999999999,"5min":202.673,"15min":223.42500000000001},"last":18.164999999999999}]},{"osd":7,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":23.655999999999999,"5min":30.402999999999999,"15min":23.09},"min":{"1min":15.859,"5min":11.272,"15min":7.4580000000000002},"max":{"1min":70.263999999999996,"5min":190.77799999999999,"15min":224.63},"last":17.344000000000001},{"interface":"front","average":{"1min":21.408999999999999,"5min":31.376999999999999,"15min":27.039999999999999},"min":{"1min":8.7430000000000003,"5min":8.7430000000000003,"15min":8.7430000000000003},"max":{"1min":74.259,"5min":203.26499999999999,"15min":220.893},"last":12.231}]}]},{"osd":5,"up_from":17,"seq":73014444474,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72604,"kb_used_data":5324,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299236,"statfs":{"total":96636764160,"available":96562417664,"internally_reserved":0,"allocated":5451776,"data_stored":2742327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":14.396000000000001,"5min":19.771999999999998,"15min":16.498999999999999},"min":{"1min":6.2709999999999999,"5min":5.6180000000000003,"15min":4.5090000000000003},"max":{"1min":72.215999999999994,"5min":115.977,"15min":350.07600000000002},"last":13.756},{"interface":"front","average":{"1min":20.567,"5min":26.533999999999999,"15min":22.433},"min":{"1min":9.1340000000000003,"5min":7.8810000000000002,"15min":6.133},"max":{"1min":81.966999999999999,"5min":128.446,"15min":351.94},"last":15.007999999999999}]},{"osd":1,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":16.960000000000001,"5min":22.443000000000001,"15min":21.648},"min":{"1min":7.9630000000000001,"5min":7.0830000000000002,"15min":6.2359999999999998},"max":{"1min":46.289000000000001,"5min":158.76400000000001,"15min":275.12},"last":14.387},{"interface":"front","average":{"1min":19.393000000000001,"5min":24.099,"15min":22.812999999999999},"min":{"1min":8.5939999999999994,"5min":8.1590000000000007,"15min":4.6520000000000001},"max":{"1min":61.463000000000001,"5min":112.78700000000001,"15min":369.73200000000003},"last":15.612}]},{"osd":2,"last update":"Tue Feb 6 22:17:21 2024","interfaces":[{"interface":"back","average":{"1min":12.752000000000001,"5min":43.362000000000002,"15min":31.550000000000001},"min":{"1min":5.7140000000000004,"5min":5.7140000000000004,"15min":5.7140000000000004},"max":{"1min":46.295000000000002,"5min":2330.5610000000001,"15min":2330.5610000000001},"last":7.3019999999999996},{"interface":"front","average":{"1min":17.791,"5min":50.844999999999999,"15min":35.262999999999998},"min":{"1min":7.3230000000000004,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":44.875999999999998,"5min":2614.4409999999998,"15min":2614.4409999999998},"last":19.096}]},{"osd":3,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":18.696000000000002,"5min":23.902000000000001,"15min":24.218},"min":{"1min":7.8049999999999997,"5min":6.125,"15min":6.125},"max":{"1min":62.171999999999997,"5min":156.80000000000001,"15min":393.55200000000002},"last":10.568},{"interface":"front","average":{"1min":14.869,"5min":19.66,"15min":21.309999999999999},"min":{"1min":6.1159999999999997,"5min":5.3819999999999997,"15min":5.3819999999999997},"max":{"1min":63.813000000000002,"5min":104.755,"15min":376.113},"last":8.3789999999999996}]},{"osd":4,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":21.885000000000002,"15min":23.100999999999999},"min":{"1min":6.5650000000000004,"5min":6.093,"15min":6.093},"max":{"1min":84.843999999999994,"5min":105.532,"15min":426.77300000000002},"last":13.170999999999999},{"interface":"front","average":{"1min":18.773,"5min":21.994,"15min":24.087},"min":{"1min":7.4870000000000001,"5min":7.3819999999999997,"15min":5.8330000000000002},"max":{"1min":50.149000000000001,"5min":90.284000000000006,"15min":419.16000000000003},"last":17.120999999999999}]},{"osd":6,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":20.283000000000001,"5min":29.465,"15min":28.431999999999999},"min":{"1min":6.9560000000000004,"5min":6.9560000000000004,"15min":6.9560000000000004},"max":{"1min":85.828000000000003,"5min":219.14099999999999,"15min":392.71600000000001},"last":12.541},{"interface":"front","average":{"1min":21.872,"5min":25.550000000000001,"15min":26.341000000000001},"min":{"1min":9.8740000000000006,"5min":7.2350000000000003,"15min":7.2350000000000003},"max":{"1min":65.367000000000004,"5min":152.69900000000001,"15min":377.31400000000002},"last":11.234999999999999}]},{"osd":7,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":19.959,"5min":25.904,"15min":25.827999999999999},"min":{"1min":9.0510000000000002,"5min":9.0510000000000002,"15min":9.0510000000000002},"max":{"1min":63.646000000000001,"5min":124.711,"15min":377.87400000000002},"last":16.288},{"interface":"front","average":{"1min":19.725000000000001,"5min":27.969999999999999,"15min":24.045999999999999},"min":{"1min":7.8940000000000001,"5min":7.8940000000000001,"15min":6.7549999999999999},"max":{"1min":48.325000000000003,"5min":152.59399999999999,"15min":338.803},"last":9.9830000000000005}]}]},{"osd":4,"up_from":17,"seq":73014444471,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72244,"kb_used_data":4968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299596,"statfs":{"total":96636764160,"available":96562786304,"internally_reserved":0,"allocated":5087232,"data_stored":2367944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.2439999999999998,"5min":17.853000000000002,"15min":16.888999999999999},"min":{"1min":4.907,"5min":4.4169999999999998,"15min":4.3899999999999997},"max":{"1min":19.225999999999999,"5min":167.291,"15min":216.86600000000001},"last":7.6379999999999999},{"interface":"front","average":{"1min":9.9450000000000003,"5min":22.707999999999998,"15min":18.631},"min":{"1min":4.742,"5min":4.6920000000000002,"15min":4.6829999999999998},"max":{"1min":23.745000000000001,"5min":231.82599999999999,"15min":287.97300000000001},"last":14.872}]},{"osd":1,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":17.431999999999999,"5min":24.254999999999999,"15min":30.890000000000001},"min":{"1min":6.2949999999999999,"5min":6.2949999999999999,"15min":5.1539999999999999},"max":{"1min":65.745999999999995,"5min":207.22399999999999,"15min":678.16999999999996},"last":9.7309999999999999},{"interface":"front","average":{"1min":15.343999999999999,"5min":23.465,"15min":32.118000000000002},"min":{"1min":8.4299999999999997,"5min":6.2599999999999998,"15min":6.2599999999999998},"max":{"1min":25.268999999999998,"5min":203.19999999999999,"15min":667.16300000000001},"last":11.487}]},{"osd":2,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":15.932,"5min":59.622999999999998,"15min":35.741999999999997},"min":{"1min":10.554,"5min":6.601,"15min":6.601},"max":{"1min":32.744999999999997,"5min":2277.7020000000002,"15min":2277.7020000000002},"last":15.648999999999999},{"interface":"front","average":{"1min":22.949999999999999,"5min":66.596999999999994,"15min":44.031999999999996},"min":{"1min":12.722,"5min":9.4559999999999995,"15min":9.4559999999999995},"max":{"1min":52.287999999999997,"5min":2279.0419999999999,"15min":2279.0419999999999},"last":17.969000000000001}]},{"osd":3,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":21.295999999999999,"5min":30.468,"15min":28.460999999999999},"min":{"1min":9.4499999999999993,"5min":6.3559999999999999,"15min":6.3559999999999999},"max":{"1min":62.805999999999997,"5min":254.292,"15min":423.20400000000001},"last":17.077999999999999},{"interface":"front","average":{"1min":15.101000000000001,"5min":31.655000000000001,"15min":24.419},"min":{"1min":6.9610000000000003,"5min":6.9610000000000003,"15min":5.5910000000000002},"max":{"1min":25.286999999999999,"5min":225.64699999999999,"15min":359.959},"last":12.189}]},{"osd":5,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":22.587,"5min":34.43,"15min":29.388999999999999},"min":{"1min":12.337999999999999,"5min":10.756,"15min":7.2999999999999998},"max":{"1min":66.715999999999994,"5min":248.70500000000001,"15min":363.36900000000003},"last":20.989000000000001},{"interface":"front","average":{"1min":15.039999999999999,"5min":28.181999999999999,"15min":27.018999999999998},"min":{"1min":6.6040000000000001,"5min":6.476,"15min":6.1550000000000002},"max":{"1min":43.223999999999997,"5min":208.02699999999999,"15min":809.58100000000002},"last":10.851000000000001}]},{"osd":6,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":17.777999999999999,"5min":24.445,"15min":26.818999999999999},"min":{"1min":7.5819999999999999,"5min":6.2930000000000001,"15min":6.2930000000000001},"max":{"1min":63.929000000000002,"5min":174.40700000000001,"15min":400.46800000000002},"last":12.803000000000001},{"interface":"front","average":{"1min":24.074999999999999,"5min":35.155000000000001,"15min":36.561},"min":{"1min":13.917,"5min":11.411,"15min":10.502000000000001},"max":{"1min":66.519999999999996,"5min":269.44200000000001,"15min":498.22300000000001},"last":20.058}]},{"osd":7,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":24.492000000000001,"5min":35.420000000000002,"15min":26.943000000000001},"min":{"1min":14.81,"5min":8.9009999999999998,"15min":6.6239999999999997},"max":{"1min":52.576999999999998,"5min":287.61599999999999,"15min":360.55599999999998},"last":18.937999999999999},{"interface":"front","average":{"1min":15.82,"5min":26.690999999999999,"15min":28.559999999999999},"min":{"1min":9.7210000000000001,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":28.702999999999999,"5min":215.26400000000001,"15min":478.649},"last":8.8460000000000001}]}]},{"osd":3,"up_from":17,"seq":73014444471,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72244,"kb_used_data":4968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299596,"statfs":{"total":96636764160,"available":96562786304,"internally_reserved":0,"allocated":5087232,"data_stored":2367944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:35 2024","interfaces":[{"interface":"back","average":{"1min":15.148999999999999,"5min":19.969999999999999,"15min":18.234000000000002},"min":{"1min":5.6100000000000003,"5min":4.5270000000000001,"15min":4.173},"max":{"1min":37.642000000000003,"5min":178.14400000000001,"15min":233.84200000000001},"last":10.285},{"interface":"front","average":{"1min":17.673999999999999,"5min":22.867000000000001,"15min":22.109999999999999},"min":{"1min":7.1299999999999999,"5min":5.3949999999999996,"15min":5.3949999999999996},"max":{"1min":45.851999999999997,"5min":224.822,"15min":224.822},"last":11.699999999999999}]},{"osd":1,"last update":"Tue Feb 6 22:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.183999999999999,"5min":19.210000000000001,"15min":21.574000000000002},"min":{"1min":6.7039999999999997,"5min":6.7039999999999997,"15min":6.7039999999999997},"max":{"1min":21.882999999999999,"5min":169.31100000000001,"15min":273.12700000000001},"last":14.042999999999999},{"interface":"front","average":{"1min":16.084,"5min":25.670000000000002,"15min":25.431999999999999},"min":{"1min":10.845000000000001,"5min":10.845000000000001,"15min":9.2880000000000003},"max":{"1min":24.265000000000001,"5min":205.495,"15min":322.70499999999998},"last":12.353}]},{"osd":2,"last update":"Tue Feb 6 22:17:35 2024","interfaces":[{"interface":"back","average":{"1min":25.116,"5min":28.814,"15min":26.242999999999999},"min":{"1min":11.129,"5min":11.129,"15min":9.1999999999999993},"max":{"1min":59.799999999999997,"5min":240.316,"15min":282.40699999999998},"last":13.276},{"interface":"front","average":{"1min":11.965,"5min":28.228999999999999,"15min":21.456},"min":{"1min":5.8979999999999997,"5min":5.8419999999999996,"15min":5.1550000000000002},"max":{"1min":39.423000000000002,"5min":998.23500000000001,"15min":998.23500000000001},"last":6.3099999999999996}]},{"osd":4,"last update":"Tue Feb 6 22:17:17 2024","interfaces":[{"interface":"back","average":{"1min":12.571999999999999,"5min":17.777000000000001,"15min":20.849},"min":{"1min":7.6459999999999999,"5min":6.1109999999999998,"15min":5.8339999999999996},"max":{"1min":25.420000000000002,"5min":162.65299999999999,"15min":189.12},"last":8.266},{"interface":"front","average":{"1min":18.823,"5min":27.978999999999999,"15min":31.401},"min":{"1min":12.393000000000001,"5min":11.710000000000001,"15min":11.045},"max":{"1min":30.169,"5min":176.35400000000001,"15min":235.315},"last":15.599}]},{"osd":5,"last update":"Tue Feb 6 22:17:35 2024","interfaces":[{"interface":"back","average":{"1min":29.649000000000001,"5min":32.82,"15min":29.718},"min":{"1min":10.526,"5min":10.087999999999999,"15min":8.9920000000000009},"max":{"1min":77.253,"5min":264.06999999999999,"15min":626.27700000000004},"last":14.845000000000001},{"interface":"front","average":{"1min":37.348999999999997,"5min":37.712000000000003,"15min":32.811999999999998},"min":{"1min":13.699,"5min":13.699,"15min":13.699},"max":{"1min":112.84699999999999,"5min":284.88299999999998,"15min":614.85400000000004},"last":16.399000000000001}]},{"osd":6,"last update":"Tue Feb 6 22:16:52 2024","interfaces":[{"interface":"back","average":{"1min":22.713999999999999,"5min":24.713999999999999,"15min":26.106999999999999},"min":{"1min":8.7240000000000002,"5min":6.4550000000000001,"15min":6.218},"max":{"1min":55.887,"5min":177.40700000000001,"15min":954.18200000000002},"last":17.332000000000001},{"interface":"front","average":{"1min":16.166,"5min":29.777000000000001,"15min":35.177},"min":{"1min":8.3729999999999993,"5min":8.3729999999999993,"15min":8.3729999999999993},"max":{"1min":40.037999999999997,"5min":209.25800000000001,"15min":954.79300000000001},"last":7.5289999999999999}]},{"osd":7,"last update":"Tue Feb 6 22:16:52 2024","interfaces":[{"interface":"back","average":{"1min":15.278,"5min":21.925000000000001,"15min":25.556999999999999},"min":{"1min":8.2970000000000006,"5min":7.3579999999999997,"15min":7.319},"max":{"1min":40.340000000000003,"5min":173.642,"15min":655.048},"last":9.3759999999999994},{"interface":"front","average":{"1min":27.821999999999999,"5min":31.497,"15min":33.704999999999998},"min":{"1min":16.033000000000001,"5min":14.802,"15min":6.9199999999999999},"max":{"1min":69.661000000000001,"5min":206.87,"15min":653.67600000000004},"last":18.169}]}]},{"osd":2,"up_from":17,"seq":73014444471,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72244,"kb_used_data":4968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299596,"statfs":{"total":96636764160,"available":96562786304,"internally_reserved":0,"allocated":5087232,"data_stored":2367944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":9.8650000000000002,"5min":40.578000000000003,"15min":27.196999999999999},"min":{"1min":5.3460000000000001,"5min":4.6050000000000004,"15min":4.3150000000000004},"max":{"1min":23.093,"5min":1631.963,"15min":1631.963},"last":13.194000000000001},{"interface":"front","average":{"1min":9.0060000000000002,"5min":21.448,"15min":21.940999999999999},"min":{"1min":4.4859999999999998,"5min":4.4859999999999998,"15min":3.9630000000000001},"max":{"1min":17.125,"5min":138.09800000000001,"15min":370.68799999999999},"last":8.4320000000000004}]},{"osd":1,"last update":"Tue Feb 6 22:17:26 2024","interfaces":[{"interface":"back","average":{"1min":10.836,"5min":52.090000000000003,"15min":32.5},"min":{"1min":6.9000000000000004,"5min":5.9340000000000002,"15min":4.492},"max":{"1min":38.810000000000002,"5min":3146.2530000000002,"15min":3146.2530000000002},"last":6.0919999999999996},{"interface":"front","average":{"1min":19.111000000000001,"5min":58.341999999999999,"15min":36.834000000000003},"min":{"1min":8.4320000000000004,"5min":6.9669999999999996,"15min":6.9020000000000001},"max":{"1min":111.504,"5min":3147.6979999999999,"15min":3147.6979999999999},"last":9.0259999999999998}]},{"osd":3,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":11.875999999999999,"5min":51.591999999999999,"15min":33.267000000000003},"min":{"1min":8.4879999999999995,"5min":7.3890000000000002,"15min":6.1239999999999997},"max":{"1min":19.702999999999999,"5min":2357.8649999999998,"15min":2357.8649999999998},"last":7.7939999999999996},{"interface":"front","average":{"1min":17.119,"5min":58.719000000000001,"15min":37.293999999999997},"min":{"1min":9.6289999999999996,"5min":9.6289999999999996,"15min":7.133},"max":{"1min":27.391999999999999,"5min":2367.828,"15min":2367.828},"last":15.157}]},{"osd":4,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":12.506,"5min":45.119,"15min":36.264000000000003},"min":{"1min":7.5949999999999998,"5min":6.5129999999999999,"15min":6.5129999999999999},"max":{"1min":17.905000000000001,"5min":1805.6099999999999,"15min":1805.6099999999999},"last":10.436999999999999},{"interface":"front","average":{"1min":15.993,"5min":22.584,"15min":26.597999999999999},"min":{"1min":10.06,"5min":6.976,"15min":6.4009999999999998},"max":{"1min":26.065999999999999,"5min":139.29900000000001,"15min":366.25099999999998},"last":11.781000000000001}]},{"osd":5,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":13.83,"5min":59.646000000000001,"15min":39.192},"min":{"1min":8.7680000000000007,"5min":7.0650000000000004,"15min":6.9610000000000003},"max":{"1min":21.802,"5min":3145.4110000000001,"15min":3145.4110000000001},"last":11.118},{"interface":"front","average":{"1min":18.478000000000002,"5min":58.215000000000003,"15min":41.988999999999997},"min":{"1min":12.645,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":28.748000000000001,"5min":2302.047,"15min":2302.047},"last":14.541}]},{"osd":6,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":13.406000000000001,"5min":31.114000000000001,"15min":30.577999999999999},"min":{"1min":10.297000000000001,"5min":7.0819999999999999,"15min":6.6479999999999997},"max":{"1min":19.422999999999998,"5min":266.53500000000003,"15min":369.185},"last":12.52},{"interface":"front","average":{"1min":19.795000000000002,"5min":60.094000000000001,"15min":42.265999999999998},"min":{"1min":14.802,"5min":9.4570000000000007,"15min":8.4930000000000003},"max":{"1min":30.856000000000002,"5min":2302.8209999999999,"15min":2302.8209999999999},"last":17.099}]},{"osd":7,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":15.435,"5min":27.277000000000001,"15min":28.007999999999999},"min":{"1min":9.0050000000000008,"5min":8.5790000000000006,"15min":8.4619999999999997},"max":{"1min":23.422000000000001,"5min":267.28300000000002,"15min":405.29899999999998},"last":13.826000000000001},{"interface":"front","average":{"1min":17.768999999999998,"5min":31.587,"15min":31.151},"min":{"1min":12.055,"5min":11.515000000000001,"15min":10.06},"max":{"1min":25.113,"5min":268.60899999999998,"15min":342.22699999999998},"last":18.021999999999998}]}]},{"osd":7,"up_from":13,"seq":55834575286,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76368,"kb_used_data":4932,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295472,"statfs":{"total":96636764160,"available":96558563328,"internally_reserved":0,"allocated":5050368,"data_stored":2344487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":11.686999999999999,"5min":23.573,"15min":18.436},"min":{"1min":5.0789999999999997,"5min":4.2880000000000003,"15min":4.2880000000000003},"max":{"1min":28.937000000000001,"5min":283.42899999999997,"15min":283.42899999999997},"last":19.643999999999998},{"interface":"front","average":{"1min":12.000999999999999,"5min":23.219999999999999,"15min":19.027999999999999},"min":{"1min":5.2949999999999999,"5min":4.3730000000000002,"15min":4.3730000000000002},"max":{"1min":33.189,"5min":279.459,"15min":387.32999999999998},"last":7.335}]},{"osd":1,"last update":"Tue Feb 6 22:17:29 2024","interfaces":[{"interface":"back","average":{"1min":13.608000000000001,"5min":19.052,"15min":22.741},"min":{"1min":6.0609999999999999,"5min":6.0609999999999999,"15min":5.577},"max":{"1min":31.280999999999999,"5min":145.70599999999999,"15min":325.41699999999997},"last":11.724},{"interface":"front","average":{"1min":13.199999999999999,"5min":17.952000000000002,"15min":21.225999999999999},"min":{"1min":6.2720000000000002,"5min":6.2720000000000002,"15min":4.7960000000000003},"max":{"1min":30.079999999999998,"5min":159.56100000000001,"15min":380.70800000000003},"last":18.518000000000001}]},{"osd":2,"last update":"Tue Feb 6 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":15.321999999999999,"5min":78.852000000000004,"15min":42.732999999999997},"min":{"1min":7.1890000000000001,"5min":6.367,"15min":5.8769999999999998},"max":{"1min":56.795000000000002,"5min":3537.9119999999998,"15min":3537.9119999999998},"last":9.8010000000000002},{"interface":"front","average":{"1min":17,"5min":81.25,"15min":44.497},"min":{"1min":9.3260000000000005,"5min":7.6790000000000003,"15min":6.6539999999999999},"max":{"1min":26.536999999999999,"5min":3538.9259999999999,"15min":3538.9259999999999},"last":21.727}]},{"osd":3,"last update":"Tue Feb 6 22:17:29 2024","interfaces":[{"interface":"back","average":{"1min":16.347999999999999,"5min":20.353999999999999,"15min":21.109999999999999},"min":{"1min":7.7990000000000004,"5min":6.8399999999999999,"15min":6.3390000000000004},"max":{"1min":57.917999999999999,"5min":186.297,"15min":317.94600000000003},"last":15.587999999999999},{"interface":"front","average":{"1min":22.106000000000002,"5min":27.684000000000001,"15min":27.027999999999999},"min":{"1min":14.65,"5min":9.8360000000000003,"15min":9.8360000000000003},"max":{"1min":58.795000000000002,"5min":201.75299999999999,"15min":325.60599999999999},"last":22.898}]},{"osd":4,"last update":"Tue Feb 6 22:17:20 2024","interfaces":[{"interface":"back","average":{"1min":20.239999999999998,"5min":22.088999999999999,"15min":22.254999999999999},"min":{"1min":9.7159999999999993,"5min":9.2370000000000001,"15min":6.9690000000000003},"max":{"1min":61.972999999999999,"5min":102.001,"15min":235.27799999999999},"last":10.444000000000001},{"interface":"front","average":{"1min":14.773999999999999,"5min":18.849,"15min":25.335000000000001},"min":{"1min":7.585,"5min":6.8440000000000003,"15min":6.8440000000000003},"max":{"1min":25.353000000000002,"5min":87.968999999999994,"15min":350.33300000000003},"last":14.77}]},{"osd":5,"last update":"Tue Feb 6 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":24.497,"5min":25.561,"15min":24.407},"min":{"1min":12.058,"5min":7.1189999999999998,"15min":7.1189999999999998},"max":{"1min":89.010999999999996,"5min":201.70400000000001,"15min":214.83500000000001},"last":20.087},{"interface":"front","average":{"1min":18.481999999999999,"5min":22.195,"15min":25.890999999999998},"min":{"1min":8.6969999999999992,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":65.813999999999993,"5min":106.07299999999999,"15min":301.78399999999999},"last":24.082999999999998}]},{"osd":6,"last update":"Tue Feb 6 22:17:29 2024","interfaces":[{"interface":"back","average":{"1min":23.632999999999999,"5min":26.312999999999999,"15min":25.606000000000002},"min":{"1min":15.241,"5min":8.3239999999999998,"15min":8.3239999999999998},"max":{"1min":60.616,"5min":203.58199999999999,"15min":235.27799999999999},"last":25.675999999999998},{"interface":"front","average":{"1min":16.300999999999998,"5min":22.77,"15min":26.331},"min":{"1min":9.3000000000000007,"5min":7.3179999999999996,"15min":6.7400000000000002},"max":{"1min":28.381,"5min":185.49799999999999,"15min":490.68299999999999},"last":13.365}]}]},{"osd":0,"up_from":13,"seq":55834575283,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72208,"kb_used_data":4932,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299632,"statfs":{"total":96636764160,"available":96562823168,"internally_reserved":0,"allocated":5050368,"data_stored":2344487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 6 22:16:50 2024","interfaces":[{"interface":"back","average":{"1min":15.423,"5min":27.477,"15min":24.606000000000002},"min":{"1min":5.4459999999999997,"5min":4.5460000000000003,"15min":4.4029999999999996},"max":{"1min":93.102999999999994,"5min":314.11000000000001,"15min":416.55599999999998},"last":6.6349999999999998},{"interface":"front","average":{"1min":11.308999999999999,"5min":21.106000000000002,"15min":23.184999999999999},"min":{"1min":5.8769999999999998,"5min":5.5389999999999997,"15min":4.5},"max":{"1min":20.073,"5min":199.238,"15min":407.44799999999998},"last":9.0109999999999992}]},{"osd":2,"last update":"Tue Feb 6 22:16:50 2024","interfaces":[{"interface":"back","average":{"1min":18.163,"5min":27.111999999999998,"15min":29.199000000000002},"min":{"1min":6.4880000000000004,"5min":6.4880000000000004,"15min":6.3799999999999999},"max":{"1min":72.644000000000005,"5min":264.48200000000003,"15min":395.14800000000002},"last":12.398999999999999},{"interface":"front","average":{"1min":15.352,"5min":27.988,"15min":25.997},"min":{"1min":5.8390000000000004,"5min":5.5259999999999998,"15min":5.383},"max":{"1min":62.317,"5min":514.94500000000005,"15min":514.94500000000005},"last":11.593999999999999}]},{"osd":3,"last update":"Tue Feb 6 22:16:50 2024","interfaces":[{"interface":"back","average":{"1min":14.545,"5min":24.606999999999999,"15min":25.003},"min":{"1min":7.633,"5min":6.351,"15min":5.8140000000000001},"max":{"1min":26.542999999999999,"5min":291.59100000000001,"15min":376.95100000000002},"last":27.548999999999999},{"interface":"front","average":{"1min":14.034000000000001,"5min":25.123000000000001,"15min":26.632999999999999},"min":{"1min":6.6139999999999999,"5min":6.6139999999999999,"15min":6.6139999999999999},"max":{"1min":26.699999999999999,"5min":262.06900000000002,"15min":395.83100000000002},"last":7.75}]},{"osd":4,"last update":"Tue Feb 6 22:17:09 2024","interfaces":[{"interface":"back","average":{"1min":16.298999999999999,"5min":31.213000000000001,"15min":28.899000000000001},"min":{"1min":9.8650000000000002,"5min":7.3609999999999998,"15min":6.6970000000000001},"max":{"1min":27.573,"5min":491.53100000000001,"15min":491.53100000000001},"last":14.371},{"interface":"front","average":{"1min":11.919,"5min":26.989999999999998,"15min":25.725000000000001},"min":{"1min":8.4060000000000006,"5min":7.2409999999999997,"15min":6.6589999999999998},"max":{"1min":18.683,"5min":490.28699999999998,"15min":490.28699999999998},"last":10.390000000000001}]},{"osd":5,"last update":"Tue Feb 6 22:16:47 2024","interfaces":[{"interface":"back","average":{"1min":21.765000000000001,"5min":30.367999999999999,"15min":29.795000000000002},"min":{"1min":9.1989999999999998,"5min":7.9969999999999999,"15min":7.9969999999999999},"max":{"1min":41.332000000000001,"5min":282.00799999999998,"15min":480.24200000000002},"last":10.941000000000001},{"interface":"front","average":{"1min":16.699000000000002,"5min":25.358000000000001,"15min":25.074999999999999},"min":{"1min":9.0739999999999998,"5min":8.9350000000000005,"15min":6.8620000000000001},"max":{"1min":27.058,"5min":193.21700000000001,"15min":347.78699999999998},"last":9.7750000000000004}]},{"osd":6,"last update":"Tue Feb 6 22:16:47 2024","interfaces":[{"interface":"back","average":{"1min":25.838999999999999,"5min":31.405999999999999,"15min":31.593},"min":{"1min":11.106999999999999,"5min":9.4789999999999992,"15min":8.9139999999999997},"max":{"1min":85.287000000000006,"5min":218.12,"15min":550.89700000000005},"last":28.219999999999999},{"interface":"front","average":{"1min":19.417999999999999,"5min":26.103000000000002,"15min":26.722999999999999},"min":{"1min":9.5619999999999994,"5min":9.1310000000000002,"15min":8.3390000000000004},"max":{"1min":29.532,"5min":175.947,"15min":486.25299999999999},"last":22.055}]},{"osd":7,"last update":"Tue Feb 6 22:17:38 2024","interfaces":[{"interface":"back","average":{"1min":24.734000000000002,"5min":33.390999999999998,"15min":30.097000000000001},"min":{"1min":13.772,"5min":11.374000000000001,"15min":9.6170000000000009},"max":{"1min":107.01000000000001,"5min":218.108,"15min":404.22800000000001},"last":22.709},{"interface":"front","average":{"1min":28.641999999999999,"5min":38.777999999999999,"15min":34.070999999999998},"min":{"1min":15.483000000000001,"5min":13.800000000000001,"15min":11.497999999999999},"max":{"1min":109.504,"5min":219.56299999999999,"15min":415.92000000000002},"last":24.742999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-06T22:17:57.642 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-06T22:17:57.741 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.740+0000 7ff66f4ba640 10 monclient: get_monmap_and_config 2024-02-06T22:17:57.741 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.740+0000 7ff66f4ba640 10 monclient: build_initial_monmap 2024-02-06T22:17:57.741 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.740+0000 7ff66f4ba640 10 monclient: monmap: 2024-02-06T22:17:57.741 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:17:57.741 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:57.742 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:17:57.741421+0000 2024-02-06T22:17:57.742 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:17:57.741421+0000 2024-02-06T22:17:57.742 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:17:57.742 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:57.742 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:17:57.742 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:17:57.742 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:17:57.742 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:17:57.742 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:17:57.742 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:57.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.740+0000 7ff66f4ba640 1 Processor -- start 2024-02-06T22:17:57.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.740+0000 7ff66f4ba640 1 -- start start 2024-02-06T22:17:57.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.740+0000 7ff66f4ba640 10 monclient: init 2024-02-06T22:17:57.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 10 monclient: _reopen_session rank -1 2024-02-06T22:17:57.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-02-06T22:17:57.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff6680b54b0 0x7ff6680a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:57.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 10 monclient(hunting): picked mon.noname-c con 0x7ff6680b54b0 addr v2:172.21.15.135:3300/0 2024-02-06T22:17:57.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff66813f830 0x7ff66813fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:57.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 10 monclient(hunting): picked mon.noname-a con 0x7ff66813f830 addr v2:172.21.15.120:3300/0 2024-02-06T22:17:57.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff668142680 0x7ff668142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:57.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 10 monclient(hunting): picked mon.noname-b con 0x7ff668142680 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:17:57.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:57.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6680ab520 con 0x7ff66813f830 2024-02-06T22:17:57.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:57.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6680b5890 con 0x7ff6680b54b0 2024-02-06T22:17:57.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:57.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6680aa450 con 0x7ff668142680 2024-02-06T22:17:57.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 10 monclient(hunting): _renew_subs 2024-02-06T22:17:57.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.741+0000 7ff66f4ba640 10 monclient(hunting): authenticate will time out at 3508.079133s 2024-02-06T22:17:57.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66da57640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff668142680 0x7ff668142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:57.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66ca55640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff66813f830 0x7ff66813fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:57.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66d256640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff6680b54b0 0x7ff6680a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:57.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66da57640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff668142680 0x7ff668142ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:58498/0 (socket says 172.21.15.120:58498) 2024-02-06T22:17:57.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66da57640 1 -- 172.21.15.120:0/3192801365 learned_addr learned my addr 172.21.15.120:0/3192801365 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:17:57.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66da57640 10 monclient(hunting): get_auth_request con 0x7ff668142680 auth_method 0 2024-02-06T22:17:57.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66da57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:57.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66da57640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:57.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66da57640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:57.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66ca55640 10 monclient(hunting): get_auth_request con 0x7ff66813f830 auth_method 0 2024-02-06T22:17:57.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66ca55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:57.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66ca55640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:57.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66ca55640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:57.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66d256640 10 monclient(hunting): get_auth_request con 0x7ff6680b54b0 auth_method 0 2024-02-06T22:17:57.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66d256640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:57.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66d256640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:57.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.742+0000 7ff66d256640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:57.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.743+0000 7ff66da57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:57.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.743+0000 7ff66da57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:57.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.743+0000 7ff66da57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:57.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.743+0000 7ff66d256640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:57.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.743+0000 7ff66d256640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:57.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.743+0000 7ff66d256640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:57.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.743+0000 7ff66ca55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:57.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.743+0000 7ff66ca55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:57.748 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.743+0000 7ff66ca55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:57.748 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.744+0000 7ff66d256640 10 monclient(hunting): handle_auth_done global_id 21010 payload 274 2024-02-06T22:17:57.748 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.744+0000 7ff66d256640 1 -- 172.21.15.120:0/3192801365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff668142680 msgr2=0x7ff668142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:17:57.748 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.744+0000 7ff66d256640 1 --2- 172.21.15.120:0/3192801365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff668142680 0x7ff668142ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.744+0000 7ff66d256640 1 -- 172.21.15.120:0/3192801365 >> v2:172.21.15.120:3300/0 conn(0x7ff66813f830 msgr2=0x7ff66813fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:17:57.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.744+0000 7ff66d256640 1 --2- 172.21.15.120:0/3192801365 >> v2:172.21.15.120:3300/0 conn(0x7ff66813f830 0x7ff66813fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.744+0000 7ff66d256640 10 monclient: _finish_hunting 0 2024-02-06T22:17:57.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.744+0000 7ff66d256640 1 monclient: found mon.noname-c 2024-02-06T22:17:57.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.744+0000 7ff66d256640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:17:57.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.744+0000 7ff66d256640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T22:17:57.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.744+0000 7ff66d256640 1 -- 172.21.15.120:0/3192801365 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6680ab720 con 0x7ff6680b54b0 2024-02-06T22:17:57.749 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.746+0000 7ff66d256640 1 --2- 172.21.15.120:0/3192801365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6680b54b0 0x7ff6680a9860 secure :-1 s=READY pgs=5084 cs=0 l=1 rev1=1 crypto rx=0x7ff66806c7b0 tx=0x7ff66402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9594fc727a41f593 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:17:57.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff65ffff640 1 -- 172.21.15.120:0/3192801365 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66402e070 con 0x7ff6680b54b0 2024-02-06T22:17:57.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff65ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:57.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff65ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T22:17:57.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff65ffff640 10 monclient: dump: 2024-02-06T22:17:57.752 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:57.752 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:57.752 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.752 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.752 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:57.752 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:57.752 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:57.752 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:57.752 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:57.752 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:57.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff65ffff640 10 monclient: _finish_auth 0 2024-02-06T22:17:57.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff65ffff640 10 monclient: _check_auth_tickets 2024-02-06T22:17:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff65ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:17:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff65ffff640 1 -- 172.21.15.120:0/3192801365 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6640315a0 con 0x7ff6680b54b0 2024-02-06T22:17:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 5 monclient: authenticate success, global_id 21010 2024-02-06T22:17:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T22:17:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff65ffff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:17:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 10 monclient: get_monmap_and_config success 2024-02-06T22:17:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 4 set_mon_vals no callback set 2024-02-06T22:17:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff65ffff640 1 -- 172.21.15.120:0/3192801365 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664031930 con 0x7ff6680b54b0 2024-02-06T22:17:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff65ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff65ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T22:17:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff65ffff640 10 monclient: dump: 2024-02-06T22:17:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.753 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.754 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:57.754 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:57.754 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:57.754 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:57.754 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:57.754 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:57.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 10 monclient: shutdown 2024-02-06T22:17:57.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:17:57.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 1 -- 172.21.15.120:0/3192801365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6680b54b0 msgr2=0x7ff6680a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:57.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 1 --2- 172.21.15.120:0/3192801365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6680b54b0 0x7ff6680a9860 secure :-1 s=READY pgs=5084 cs=0 l=1 rev1=1 crypto rx=0x7ff66806c7b0 tx=0x7ff66402b840 comp rx=0 tx=0).stop 2024-02-06T22:17:57.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 1 -- 172.21.15.120:0/3192801365 shutdown_connections 2024-02-06T22:17:57.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 1 --2- 172.21.15.120:0/3192801365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff668142680 0x7ff668142ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 1 --2- 172.21.15.120:0/3192801365 >> v2:172.21.15.120:3300/0 conn(0x7ff66813f830 0x7ff66813fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.757 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 1 --2- 172.21.15.120:0/3192801365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6680b54b0 0x7ff6680a9860 unknown :-1 s=CLOSED pgs=5084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 1 -- 172.21.15.120:0/3192801365 >> 172.21.15.120:0/3192801365 conn(0x7ff6680b8660 msgr2=0x7ff6680b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:17:57.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 1 -- 172.21.15.120:0/3192801365 shutdown_connections 2024-02-06T22:17:57.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 1 -- 172.21.15.120:0/3192801365 wait complete. 2024-02-06T22:17:57.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 10 monclient: build_initial_monmap 2024-02-06T22:17:57.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.751+0000 7ff66f4ba640 10 monclient: monmap: 2024-02-06T22:17:57.758 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:17:57.758 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:17:57.758 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:17:57.758 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:17:57.758 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:17:57.758 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:57.758 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:17:57.758 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:17:57.758 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:17:57.758 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:57.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 1 librados: starting msgr at 2024-02-06T22:17:57.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 1 librados: starting objecter 2024-02-06T22:17:57.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 1 Processor -- start 2024-02-06T22:17:57.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 1 -- start start 2024-02-06T22:17:57.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 1 librados: setting wanted keys 2024-02-06T22:17:57.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 1 librados: calling monclient init 2024-02-06T22:17:57.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 10 monclient: init 2024-02-06T22:17:57.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 10 monclient: _reopen_session rank -1 2024-02-06T22:17:57.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-02-06T22:17:57.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6680b54b0 0x7ff6681db190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:57.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 10 monclient(hunting): picked mon.noname-a con 0x7ff6680b54b0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:17:57.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff66813f830 0x7ff6681db6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:57.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 10 monclient(hunting): picked mon.noname-c con 0x7ff66813f830 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:17:57.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff668142680 0x7ff66813e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:57.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 10 monclient(hunting): picked mon.noname-b con 0x7ff668142680 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:17:57.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:57.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6681dad00 con 0x7ff6680b54b0 2024-02-06T22:17:57.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:57.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66813d800 con 0x7ff668142680 2024-02-06T22:17:57.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:57.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66813d2c0 con 0x7ff66813f830 2024-02-06T22:17:57.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 10 monclient(hunting): _renew_subs 2024-02-06T22:17:57.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.752+0000 7ff66f4ba640 10 monclient(hunting): authenticate will time out at 3508.090362s 2024-02-06T22:17:57.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.753+0000 7ff66d256640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6680b54b0 0x7ff6681db190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:57.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.753+0000 7ff66ca55640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff66813f830 0x7ff6681db6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:57.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.753+0000 7ff66ca55640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff66813f830 0x7ff6681db6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:58512/0 (socket says 172.21.15.120:58512) 2024-02-06T22:17:57.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.753+0000 7ff66ca55640 1 -- 172.21.15.120:0/178553815 learned_addr learned my addr 172.21.15.120:0/178553815 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:17:57.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.753+0000 7ff66ca55640 10 monclient(hunting): get_auth_request con 0x7ff66813f830 auth_method 0 2024-02-06T22:17:57.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.753+0000 7ff66ca55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:57.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.753+0000 7ff66ca55640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:57.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.753+0000 7ff66ca55640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:57.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.754+0000 7ff66d256640 10 monclient(hunting): get_auth_request con 0x7ff6680b54b0 auth_method 0 2024-02-06T22:17:57.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.754+0000 7ff66d256640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:57.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.754+0000 7ff66d256640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:57.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.754+0000 7ff66d256640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:57.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.754+0000 7ff66d256640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:57.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.754+0000 7ff66d256640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:57.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.754+0000 7ff66d256640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:57.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.754+0000 7ff66da57640 1 --2- 172.21.15.120:0/178553815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff668142680 0x7ff66813e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:57.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.755+0000 7ff66da57640 10 monclient(hunting): get_auth_request con 0x7ff668142680 auth_method 0 2024-02-06T22:17:57.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.755+0000 7ff66da57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:57.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.755+0000 7ff66da57640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:57.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.755+0000 7ff66da57640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:57.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.756+0000 7ff66da57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:57.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.756+0000 7ff66da57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:57.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.756+0000 7ff66da57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:57.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.756+0000 7ff66ca55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:57.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.756+0000 7ff66ca55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:57.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.756+0000 7ff66ca55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:57.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.757+0000 7ff66d256640 10 monclient(hunting): handle_auth_done global_id 16491 payload 824 2024-02-06T22:17:57.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.757+0000 7ff66d256640 1 -- 172.21.15.120:0/178553815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff66813f830 msgr2=0x7ff6681db6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:57.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.757+0000 7ff66d256640 1 --2- 172.21.15.120:0/178553815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff66813f830 0x7ff6681db6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.757+0000 7ff66d256640 1 -- 172.21.15.120:0/178553815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff668142680 msgr2=0x7ff66813e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:57.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.757+0000 7ff66d256640 1 --2- 172.21.15.120:0/178553815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff668142680 0x7ff66813e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:57.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.757+0000 7ff66d256640 10 monclient: _finish_hunting 0 2024-02-06T22:17:57.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.757+0000 7ff66d256640 1 monclient: found mon.noname-a 2024-02-06T22:17:57.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.757+0000 7ff66d256640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:17:57.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.757+0000 7ff66d256640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:17:57.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.757+0000 7ff66d256640 1 -- 172.21.15.120:0/178553815 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff664009400 con 0x7ff6680b54b0 2024-02-06T22:17:57.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.760+0000 7ff66d256640 1 --2- 172.21.15.120:0/178553815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6680b54b0 0x7ff6681db190 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7ff66806deb0 tx=0x7ff6640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:17:57.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.760+0000 7ff65dffb640 10 client.?.objecter ms_handle_connect 0x7ff6680b54b0 2024-02-06T22:17:57.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.760+0000 7ff65dffb640 10 client.?.objecter resend_mon_ops 2024-02-06T22:17:57.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff65dffb640 1 -- 172.21.15.120:0/178553815 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664032630 con 0x7ff6680b54b0 2024-02-06T22:17:57.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff65dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:57.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff65dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:17:57.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff65dffb640 10 monclient: dump: 2024-02-06T22:17:57.768 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:57.768 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:57.768 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.768 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.768 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:57.768 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:57.768 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:57.769 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:57.769 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:57.769 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:57.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff65dffb640 10 monclient: _finish_auth 0 2024-02-06T22:17:57.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff65dffb640 10 monclient: _check_auth_tickets 2024-02-06T22:17:57.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff65dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:17:57.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff65dffb640 1 -- 172.21.15.120:0/178553815 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6640327b0 con 0x7ff6680b54b0 2024-02-06T22:17:57.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff65dffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:17:57.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff65dffb640 1 -- 172.21.15.120:0/178553815 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664032b60 con 0x7ff6680b54b0 2024-02-06T22:17:57.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff66f4ba640 5 monclient: authenticate success, global_id 16491 2024-02-06T22:17:57.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff66f4ba640 10 monclient: _renew_subs 2024-02-06T22:17:57.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff66f4ba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:17:57.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff66f4ba640 1 -- 172.21.15.120:0/178553815 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff66813d480 con 0x7ff6680b54b0 2024-02-06T22:17:57.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff65f7fe640 4 set_mon_vals no callback set 2024-02-06T22:17:57.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff65dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:57.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff65dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:17:57.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff65dffb640 10 monclient: dump: 2024-02-06T22:17:57.770 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:57.770 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:57.770 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.770 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:57.770 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:57.770 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:57.770 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:57.771 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:57.771 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:57.771 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:57.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff66f4ba640 10 client.16491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:17:57.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff66f4ba640 10 monclient: _renew_subs 2024-02-06T22:17:57.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff66f4ba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:17:57.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff66f4ba640 1 -- 172.21.15.120:0/178553815 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6680b6c00 con 0x7ff6680b54b0 2024-02-06T22:17:57.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.766+0000 7ff66f4ba640 1 librados: init done 2024-02-06T22:17:57.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.767+0000 7ff66f4ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T22:17:57.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.767+0000 7ff66f4ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:17:57.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.767+0000 7ff66f4ba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:17:57.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:57.767+0000 7ff66f4ba640 1 -- 172.21.15.120:0/178553815 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff66804ef90 con 0x7ff6680b54b0 2024-02-06T22:17:58.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.015+0000 7ff65dffb640 1 -- 172.21.15.120:0/178553815 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff66404e3f0 con 0x7ff6680b54b0 2024-02-06T22:17:58.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.015+0000 7ff65dffb640 1 --2- 172.21.15.120:0/178553815 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff64c06d1f0 0x7ff64c06f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:58.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.016+0000 7ff66ca55640 1 --2- 172.21.15.120:0/178553815 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff64c06d1f0 0x7ff64c06f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:58.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.016+0000 7ff66ca55640 10 monclient: get_auth_request con 0x7ff64c06d1f0 auth_method 0 2024-02-06T22:17:58.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.016+0000 7ff66ca55640 1 --2- 172.21.15.120:0/178553815 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff64c06d1f0 0x7ff64c06f6b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff668141d30 tx=0x7ff658009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:17:58.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.016+0000 7ff65dffb640 10 client.16491.objecter ms_handle_connect 0x7ff64c06d1f0 2024-02-06T22:17:58.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.033+0000 7ff65dffb640 1 -- 172.21.15.120:0/178553815 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(1345..1345 src has 1047..1345) v4 ==== 6389+0+0 (secure 0 0 0) 0x7ff66407acd0 con 0x7ff6680b54b0 2024-02-06T22:17:58.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.033+0000 7ff65dffb640 10 client.16491.objecter ms_dispatch 0x7ff668000b90 osd_map(1345..1345 src has 1047..1345) v4 2024-02-06T22:17:58.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.033+0000 7ff65dffb640 3 client.16491.objecter handle_osd_map got epochs [1345,1345] > 0 2024-02-06T22:17:58.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.033+0000 7ff65dffb640 3 client.16491.objecter handle_osd_map decoding full epoch 1345 2024-02-06T22:17:58.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.033+0000 7ff65dffb640 20 client.16491.objecter dump_active .. 0 homeless 2024-02-06T22:17:58.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.033+0000 7ff65dffb640 1 -- 172.21.15.120:0/178553815 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff664076f70 con 0x7ff6680b54b0 2024-02-06T22:17:58.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.033+0000 7ff65dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:17:58.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.033+0000 7ff65dffb640 10 monclient: _finish_command 1 = system:0 2024-02-06T22:17:58.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.205+0000 7ff66f4ba640 1 -- 172.21.15.120:0/178553815 --> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff6681401b0 con 0x7ff64c06d1f0 2024-02-06T22:17:58.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.212+0000 7ff65dffb640 1 -- 172.21.15.120:0/178553815 <== mgr.4100 v2:172.21.15.135:6800/315664144 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151689 (secure 0 0 0) 0x7ff658011af0 con 0x7ff64c06d1f0 2024-02-06T22:17:58.214 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T22:17:58.215 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2024-02-06T22:17:58.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 10 librados: watch_flush enter 2024-02-06T22:17:58.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 10 librados: watch_flush exit 2024-02-06T22:17:58.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 20 client.16491.objecter shutdown clearing up homeless session... 2024-02-06T22:17:58.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 10 client.16491.objecter successfully canceled tick 2024-02-06T22:17:58.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 1 -- 172.21.15.120:0/178553815 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff64c06d1f0 msgr2=0x7ff64c06f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:58.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 1 --2- 172.21.15.120:0/178553815 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff64c06d1f0 0x7ff64c06f6b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff668141d30 tx=0x7ff658009190 comp rx=0 tx=0).stop 2024-02-06T22:17:58.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 10 monclient: shutdown 2024-02-06T22:17:58.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:17:58.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 1 -- 172.21.15.120:0/178553815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6680b54b0 msgr2=0x7ff6681db190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:58.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 1 --2- 172.21.15.120:0/178553815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6680b54b0 0x7ff6681db190 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7ff66806deb0 tx=0x7ff6640046c0 comp rx=0 tx=0).stop 2024-02-06T22:17:58.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 1 -- 172.21.15.120:0/178553815 shutdown_connections 2024-02-06T22:17:58.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 1 --2- 172.21.15.120:0/178553815 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ff64c06d1f0 0x7ff64c06f6b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:58.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 1 --2- 172.21.15.120:0/178553815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff668142680 0x7ff66813e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:58.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 1 --2- 172.21.15.120:0/178553815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff66813f830 0x7ff6681db6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:58.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 1 --2- 172.21.15.120:0/178553815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6680b54b0 0x7ff6681db190 unknown :-1 s=CLOSED pgs=3343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:58.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 1 -- 172.21.15.120:0/178553815 >> 172.21.15.120:0/178553815 conn(0x7ff6680b8660 msgr2=0x7ff6680a95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:17:58.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 1 -- 172.21.15.120:0/178553815 shutdown_connections 2024-02-06T22:17:58.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 1 -- 172.21.15.120:0/178553815 wait complete. 2024-02-06T22:17:58.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.214+0000 7ff66f4ba640 1 librados: shutdown 2024-02-06T22:17:58.227 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":2447,"stamp":"2024-02-06T22:17:56.396453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":582792,"kb_used_data":40416,"kb_used_omap":12,"kb_used_meta":542259,"kb_avail":754391928,"statfs":{"total":773094113280,"available":772497334272,"internally_reserved":0,"allocated":41385984,"data_stored":19661387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":555273776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":135,"apply_latency_ms":135,"commit_latency_ns":135000000,"apply_latency_ns":135000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.058711"},"pg_stats":[{"pgid":"12.1c","version":"0'0","reported_seq":2582,"reported_epoch":1340,"state":"active+clean","last_fresh":"2024-02-06T22:17:51.319476+0000","last_change":"2024-02-06T21:42:50.083729+0000","last_active":"2024-02-06T22:17:51.319476+0000","last_peered":"2024-02-06T22:17:51.319476+0000","last_clean":"2024-02-06T22:17:51.319476+0000","last_became_active":"2024-02-06T21:42:50.078423+0000","last_became_peered":"2024-02-06T21:42:50.078423+0000","last_unstale":"2024-02-06T22:17:51.319476+0000","last_undegraded":"2024-02-06T22:17:51.319476+0000","last_fullsized":"2024-02-06T22:17:51.319476+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:44:16.757427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":2580,"reported_epoch":1343,"state":"active+clean","last_fresh":"2024-02-06T22:17:54.405293+0000","last_change":"2024-02-06T21:43:19.779411+0000","last_active":"2024-02-06T22:17:54.405293+0000","last_peered":"2024-02-06T22:17:54.405293+0000","last_clean":"2024-02-06T22:17:54.405293+0000","last_became_active":"2024-02-06T21:43:19.773686+0000","last_became_peered":"2024-02-06T21:43:19.773686+0000","last_unstale":"2024-02-06T22:17:54.405293+0000","last_undegraded":"2024-02-06T22:17:54.405293+0000","last_fullsized":"2024-02-06T22:17:54.405293+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:54:24.026659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":2585,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.382909+0000","last_change":"2024-02-06T21:42:53.308445+0000","last_active":"2024-02-06T22:17:53.382909+0000","last_peered":"2024-02-06T22:17:53.382909+0000","last_clean":"2024-02-06T22:17:53.382909+0000","last_became_active":"2024-02-06T21:42:53.291194+0000","last_became_peered":"2024-02-06T21:42:53.291194+0000","last_unstale":"2024-02-06T22:17:53.382909+0000","last_undegraded":"2024-02-06T22:17:53.382909+0000","last_fullsized":"2024-02-06T22:17:53.382909+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T00:08:35.742342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":2575,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.511147+0000","last_change":"2024-02-06T21:42:45.080259+0000","last_active":"2024-02-06T22:17:53.511147+0000","last_peered":"2024-02-06T22:17:53.511147+0000","last_clean":"2024-02-06T22:17:53.511147+0000","last_became_active":"2024-02-06T21:42:45.065933+0000","last_became_peered":"2024-02-06T21:42:45.065933+0000","last_unstale":"2024-02-06T22:17:53.511147+0000","last_undegraded":"2024-02-06T22:17:53.511147+0000","last_fullsized":"2024-02-06T22:17:53.511147+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:44:14.566236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":2566,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.510473+0000","last_change":"2024-02-06T21:42:58.554516+0000","last_active":"2024-02-06T22:17:50.510473+0000","last_peered":"2024-02-06T22:17:50.510473+0000","last_clean":"2024-02-06T22:17:50.510473+0000","last_became_active":"2024-02-06T21:42:58.548361+0000","last_became_peered":"2024-02-06T21:42:58.548361+0000","last_unstale":"2024-02-06T22:17:50.510473+0000","last_undegraded":"2024-02-06T22:17:50.510473+0000","last_fullsized":"2024-02-06T22:17:50.510473+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T07:18:58.848738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"124'7","reported_seq":2587,"reported_epoch":1341,"state":"active+clean","last_fresh":"2024-02-06T22:17:52.364003+0000","last_change":"2024-02-06T21:43:15.527724+0000","last_active":"2024-02-06T22:17:52.364003+0000","last_peered":"2024-02-06T22:17:52.364003+0000","last_clean":"2024-02-06T22:17:52.364003+0000","last_became_active":"2024-02-06T21:43:15.497081+0000","last_became_peered":"2024-02-06T21:43:15.497081+0000","last_unstale":"2024-02-06T22:17:52.364003+0000","last_undegraded":"2024-02-06T22:17:52.364003+0000","last_fullsized":"2024-02-06T22:17:52.364003+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:53:37.062280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":2586,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.365646+0000","last_change":"2024-02-06T21:43:13.230133+0000","last_active":"2024-02-06T22:17:53.365646+0000","last_peered":"2024-02-06T22:17:53.365646+0000","last_clean":"2024-02-06T22:17:53.365646+0000","last_became_active":"2024-02-06T21:43:13.108638+0000","last_became_peered":"2024-02-06T21:43:13.108638+0000","last_unstale":"2024-02-06T22:17:53.365646+0000","last_undegraded":"2024-02-06T22:17:53.365646+0000","last_fullsized":"2024-02-06T22:17:53.365646+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:24:54.727933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":2593,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.377205+0000","last_change":"2024-02-06T21:43:06.276185+0000","last_active":"2024-02-06T22:17:53.377205+0000","last_peered":"2024-02-06T22:17:53.377205+0000","last_clean":"2024-02-06T22:17:53.377205+0000","last_became_active":"2024-02-06T21:43:06.270947+0000","last_became_peered":"2024-02-06T21:43:06.270947+0000","last_unstale":"2024-02-06T22:17:53.377205+0000","last_undegraded":"2024-02-06T22:17:53.377205+0000","last_fullsized":"2024-02-06T22:17:53.377205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:27:26.086535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":2586,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.379590+0000","last_change":"2024-02-06T21:42:53.563677+0000","last_active":"2024-02-06T22:17:53.379590+0000","last_peered":"2024-02-06T22:17:53.379590+0000","last_clean":"2024-02-06T22:17:53.379590+0000","last_became_active":"2024-02-06T21:42:53.535059+0000","last_became_peered":"2024-02-06T21:42:53.535059+0000","last_unstale":"2024-02-06T22:17:53.379590+0000","last_undegraded":"2024-02-06T22:17:53.379590+0000","last_fullsized":"2024-02-06T22:17:53.379590+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T06:13:12.610796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":2568,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.521205+0000","last_change":"2024-02-06T21:42:46.177846+0000","last_active":"2024-02-06T22:17:50.521205+0000","last_peered":"2024-02-06T22:17:50.521205+0000","last_clean":"2024-02-06T22:17:50.521205+0000","last_became_active":"2024-02-06T21:42:46.168640+0000","last_became_peered":"2024-02-06T21:42:46.168640+0000","last_unstale":"2024-02-06T22:17:50.521205+0000","last_undegraded":"2024-02-06T22:17:50.521205+0000","last_fullsized":"2024-02-06T22:17:50.521205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:37:19.536133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":2572,"reported_epoch":1343,"state":"active+clean","last_fresh":"2024-02-06T22:17:54.417919+0000","last_change":"2024-02-06T21:43:27.891680+0000","last_active":"2024-02-06T22:17:54.417919+0000","last_peered":"2024-02-06T22:17:54.417919+0000","last_clean":"2024-02-06T22:17:54.417919+0000","last_became_active":"2024-02-06T21:43:27.883526+0000","last_became_peered":"2024-02-06T21:43:27.883526+0000","last_unstale":"2024-02-06T22:17:54.417919+0000","last_undegraded":"2024-02-06T22:17:54.417919+0000","last_fullsized":"2024-02-06T22:17:54.417919+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:21:41.402059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":2570,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.530267+0000","last_change":"2024-02-06T21:42:45.882433+0000","last_active":"2024-02-06T22:17:50.530267+0000","last_peered":"2024-02-06T22:17:50.530267+0000","last_clean":"2024-02-06T22:17:50.530267+0000","last_became_active":"2024-02-06T21:42:45.868987+0000","last_became_peered":"2024-02-06T21:42:45.868987+0000","last_unstale":"2024-02-06T22:17:50.530267+0000","last_undegraded":"2024-02-06T22:17:50.530267+0000","last_fullsized":"2024-02-06T22:17:50.530267+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T00:26:41.132150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":2572,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.498068+0000","last_change":"2024-02-06T21:42:59.955119+0000","last_active":"2024-02-06T22:17:53.498068+0000","last_peered":"2024-02-06T22:17:53.498068+0000","last_clean":"2024-02-06T22:17:53.498068+0000","last_became_active":"2024-02-06T21:42:59.949344+0000","last_became_peered":"2024-02-06T21:42:59.949344+0000","last_unstale":"2024-02-06T22:17:53.498068+0000","last_undegraded":"2024-02-06T22:17:53.498068+0000","last_fullsized":"2024-02-06T22:17:53.498068+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T06:07:53.514679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'76","reported_seq":2694,"reported_epoch":1341,"state":"active+clean","last_fresh":"2024-02-06T22:17:52.355749+0000","last_change":"2024-02-06T21:41:31.339576+0000","last_active":"2024-02-06T22:17:52.355749+0000","last_peered":"2024-02-06T22:17:52.355749+0000","last_clean":"2024-02-06T22:17:52.355749+0000","last_became_active":"2024-02-06T21:41:31.305766+0000","last_became_peered":"2024-02-06T21:41:31.305766+0000","last_unstale":"2024-02-06T22:17:52.355749+0000","last_undegraded":"2024-02-06T22:17:52.355749+0000","last_fullsized":"2024-02-06T22:17:52.355749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:20.409569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:20.409569+0000","last_clean_scrub_stamp":"2024-02-06T21:41:20.409569+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:24:57.767580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2586,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.504352+0000","last_change":"2024-02-06T21:41:36.168152+0000","last_active":"2024-02-06T22:17:53.504352+0000","last_peered":"2024-02-06T22:17:53.504352+0000","last_clean":"2024-02-06T22:17:53.504352+0000","last_became_active":"2024-02-06T21:41:36.162988+0000","last_became_peered":"2024-02-06T21:41:36.162988+0000","last_unstale":"2024-02-06T22:17:53.504352+0000","last_undegraded":"2024-02-06T22:17:53.504352+0000","last_fullsized":"2024-02-06T22:17:53.504352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T00:47:41.526670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":2569,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.526919+0000","last_change":"2024-02-06T21:42:56.031740+0000","last_active":"2024-02-06T22:17:50.526919+0000","last_peered":"2024-02-06T22:17:50.526919+0000","last_clean":"2024-02-06T22:17:50.526919+0000","last_became_active":"2024-02-06T21:42:56.023045+0000","last_became_peered":"2024-02-06T21:42:56.023045+0000","last_unstale":"2024-02-06T22:17:50.526919+0000","last_undegraded":"2024-02-06T22:17:50.526919+0000","last_fullsized":"2024-02-06T22:17:50.526919+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:19:09.477603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2587,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.467482+0000","last_change":"2024-02-06T21:41:35.688242+0000","last_active":"2024-02-06T22:17:53.467482+0000","last_peered":"2024-02-06T22:17:53.467482+0000","last_clean":"2024-02-06T22:17:53.467482+0000","last_became_active":"2024-02-06T21:41:35.642164+0000","last_became_peered":"2024-02-06T21:41:35.642164+0000","last_unstale":"2024-02-06T22:17:53.467482+0000","last_undegraded":"2024-02-06T22:17:53.467482+0000","last_fullsized":"2024-02-06T22:17:53.467482+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:50:03.838525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":2569,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.512725+0000","last_change":"2024-02-06T21:43:05.426147+0000","last_active":"2024-02-06T22:17:53.512725+0000","last_peered":"2024-02-06T22:17:53.512725+0000","last_clean":"2024-02-06T22:17:53.512725+0000","last_became_active":"2024-02-06T21:43:05.414230+0000","last_became_peered":"2024-02-06T21:43:05.414230+0000","last_unstale":"2024-02-06T22:17:53.512725+0000","last_undegraded":"2024-02-06T22:17:53.512725+0000","last_fullsized":"2024-02-06T22:17:53.512725+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:04:00.128718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2581,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.508463+0000","last_change":"2024-02-06T21:41:35.636555+0000","last_active":"2024-02-06T22:17:50.508463+0000","last_peered":"2024-02-06T22:17:50.508463+0000","last_clean":"2024-02-06T22:17:50.508463+0000","last_became_active":"2024-02-06T21:41:35.603135+0000","last_became_peered":"2024-02-06T21:41:35.603135+0000","last_unstale":"2024-02-06T22:17:50.508463+0000","last_undegraded":"2024-02-06T22:17:50.508463+0000","last_fullsized":"2024-02-06T22:17:50.508463+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T03:13:14.699865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":2570,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.522055+0000","last_change":"2024-02-06T21:42:49.418971+0000","last_active":"2024-02-06T22:17:50.522055+0000","last_peered":"2024-02-06T22:17:50.522055+0000","last_clean":"2024-02-06T22:17:50.522055+0000","last_became_active":"2024-02-06T21:42:49.409862+0000","last_became_peered":"2024-02-06T21:42:49.409862+0000","last_unstale":"2024-02-06T22:17:50.522055+0000","last_undegraded":"2024-02-06T22:17:50.522055+0000","last_fullsized":"2024-02-06T22:17:50.522055+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:08:26.530828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2587,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.487524+0000","last_change":"2024-02-06T21:41:35.892547+0000","last_active":"2024-02-06T22:17:53.487524+0000","last_peered":"2024-02-06T22:17:53.487524+0000","last_clean":"2024-02-06T22:17:53.487524+0000","last_became_active":"2024-02-06T21:41:35.877657+0000","last_became_peered":"2024-02-06T21:41:35.877657+0000","last_unstale":"2024-02-06T22:17:53.487524+0000","last_undegraded":"2024-02-06T22:17:53.487524+0000","last_fullsized":"2024-02-06T22:17:53.487524+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:23:28.851375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":2593,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.383234+0000","last_change":"2024-02-06T21:42:56.499357+0000","last_active":"2024-02-06T22:17:53.383234+0000","last_peered":"2024-02-06T22:17:53.383234+0000","last_clean":"2024-02-06T22:17:53.383234+0000","last_became_active":"2024-02-06T21:42:56.473273+0000","last_became_peered":"2024-02-06T21:42:56.473273+0000","last_unstale":"2024-02-06T22:17:53.383234+0000","last_undegraded":"2024-02-06T22:17:53.383234+0000","last_fullsized":"2024-02-06T22:17:53.383234+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:45:45.452902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2594,"reported_epoch":1340,"state":"active+clean","last_fresh":"2024-02-06T22:17:51.321473+0000","last_change":"2024-02-06T21:41:35.918723+0000","last_active":"2024-02-06T22:17:51.321473+0000","last_peered":"2024-02-06T22:17:51.321473+0000","last_clean":"2024-02-06T22:17:51.321473+0000","last_became_active":"2024-02-06T21:41:35.891748+0000","last_became_peered":"2024-02-06T21:41:35.891748+0000","last_unstale":"2024-02-06T22:17:51.321473+0000","last_undegraded":"2024-02-06T22:17:51.321473+0000","last_fullsized":"2024-02-06T22:17:51.321473+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T03:59:11.081636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":2570,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.510479+0000","last_change":"2024-02-06T21:42:49.677946+0000","last_active":"2024-02-06T22:17:50.510479+0000","last_peered":"2024-02-06T22:17:50.510479+0000","last_clean":"2024-02-06T22:17:50.510479+0000","last_became_active":"2024-02-06T21:42:49.671861+0000","last_became_peered":"2024-02-06T21:42:49.671861+0000","last_unstale":"2024-02-06T22:17:50.510479+0000","last_undegraded":"2024-02-06T22:17:50.510479+0000","last_fullsized":"2024-02-06T22:17:50.510479+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:43:48.362335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2586,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.499796+0000","last_change":"2024-02-06T21:41:35.766413+0000","last_active":"2024-02-06T22:17:53.499796+0000","last_peered":"2024-02-06T22:17:53.499796+0000","last_clean":"2024-02-06T22:17:53.499796+0000","last_became_active":"2024-02-06T21:41:35.761324+0000","last_became_peered":"2024-02-06T21:41:35.761324+0000","last_unstale":"2024-02-06T22:17:53.499796+0000","last_undegraded":"2024-02-06T22:17:53.499796+0000","last_fullsized":"2024-02-06T22:17:53.499796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:28:07.018583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":2571,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.512477+0000","last_change":"2024-02-06T21:42:44.700963+0000","last_active":"2024-02-06T22:17:50.512477+0000","last_peered":"2024-02-06T22:17:50.512477+0000","last_clean":"2024-02-06T22:17:50.512477+0000","last_became_active":"2024-02-06T21:42:44.691217+0000","last_became_peered":"2024-02-06T21:42:44.691217+0000","last_unstale":"2024-02-06T22:17:50.512477+0000","last_undegraded":"2024-02-06T22:17:50.512477+0000","last_fullsized":"2024-02-06T22:17:50.512477+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:15:55.323353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2609,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.390113+0000","last_change":"2024-02-06T21:41:36.193098+0000","last_active":"2024-02-06T22:17:53.390113+0000","last_peered":"2024-02-06T22:17:53.390113+0000","last_clean":"2024-02-06T22:17:53.390113+0000","last_became_active":"2024-02-06T21:41:36.160648+0000","last_became_peered":"2024-02-06T21:41:36.160648+0000","last_unstale":"2024-02-06T22:17:53.390113+0000","last_undegraded":"2024-02-06T22:17:53.390113+0000","last_fullsized":"2024-02-06T22:17:53.390113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T09:01:03.161738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":2582,"reported_epoch":1343,"state":"active+clean","last_fresh":"2024-02-06T22:17:54.409582+0000","last_change":"2024-02-06T21:42:52.618077+0000","last_active":"2024-02-06T22:17:54.409582+0000","last_peered":"2024-02-06T22:17:54.409582+0000","last_clean":"2024-02-06T22:17:54.409582+0000","last_became_active":"2024-02-06T21:42:52.611794+0000","last_became_peered":"2024-02-06T21:42:52.611794+0000","last_unstale":"2024-02-06T22:17:54.409582+0000","last_undegraded":"2024-02-06T22:17:54.409582+0000","last_fullsized":"2024-02-06T22:17:54.409582+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:25:35.938955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2583,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.514531+0000","last_change":"2024-02-06T21:41:36.144443+0000","last_active":"2024-02-06T22:17:53.514531+0000","last_peered":"2024-02-06T22:17:53.514531+0000","last_clean":"2024-02-06T22:17:53.514531+0000","last_became_active":"2024-02-06T21:41:36.139312+0000","last_became_peered":"2024-02-06T21:41:36.139312+0000","last_unstale":"2024-02-06T22:17:53.514531+0000","last_undegraded":"2024-02-06T22:17:53.514531+0000","last_fullsized":"2024-02-06T22:17:53.514531+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:47:25.348858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":2567,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.517568+0000","last_change":"2024-02-06T21:42:49.714094+0000","last_active":"2024-02-06T22:17:50.517568+0000","last_peered":"2024-02-06T22:17:50.517568+0000","last_clean":"2024-02-06T22:17:50.517568+0000","last_became_active":"2024-02-06T21:42:49.708820+0000","last_became_peered":"2024-02-06T21:42:49.708820+0000","last_unstale":"2024-02-06T22:17:50.517568+0000","last_undegraded":"2024-02-06T22:17:50.517568+0000","last_fullsized":"2024-02-06T22:17:50.517568+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:55:54.917799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":2584,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.362656+0000","last_change":"2024-02-06T21:43:13.244098+0000","last_active":"2024-02-06T22:17:53.362656+0000","last_peered":"2024-02-06T22:17:53.362656+0000","last_clean":"2024-02-06T22:17:53.362656+0000","last_became_active":"2024-02-06T21:43:13.123786+0000","last_became_peered":"2024-02-06T21:43:13.123786+0000","last_unstale":"2024-02-06T22:17:53.362656+0000","last_undegraded":"2024-02-06T22:17:53.362656+0000","last_fullsized":"2024-02-06T22:17:53.362656+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T02:53:35.916469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":2565,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.506337+0000","last_change":"2024-02-06T21:42:56.872139+0000","last_active":"2024-02-06T22:17:50.506337+0000","last_peered":"2024-02-06T22:17:50.506337+0000","last_clean":"2024-02-06T22:17:50.506337+0000","last_became_active":"2024-02-06T21:42:56.865960+0000","last_became_peered":"2024-02-06T21:42:56.865960+0000","last_unstale":"2024-02-06T22:17:50.506337+0000","last_undegraded":"2024-02-06T22:17:50.506337+0000","last_fullsized":"2024-02-06T22:17:50.506337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:13:16.665309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":2569,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.514243+0000","last_change":"2024-02-06T21:42:45.712485+0000","last_active":"2024-02-06T22:17:50.514243+0000","last_peered":"2024-02-06T22:17:50.514243+0000","last_clean":"2024-02-06T22:17:50.514243+0000","last_became_active":"2024-02-06T21:42:45.706432+0000","last_became_peered":"2024-02-06T21:42:45.706432+0000","last_unstale":"2024-02-06T22:17:50.514243+0000","last_undegraded":"2024-02-06T22:17:50.514243+0000","last_fullsized":"2024-02-06T22:17:50.514243+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:46:59.377729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":2569,"reported_epoch":1341,"state":"active+clean","last_fresh":"2024-02-06T22:17:52.368275+0000","last_change":"2024-02-06T21:43:08.879863+0000","last_active":"2024-02-06T22:17:52.368275+0000","last_peered":"2024-02-06T22:17:52.368275+0000","last_clean":"2024-02-06T22:17:52.368275+0000","last_became_active":"2024-02-06T21:43:08.871254+0000","last_became_peered":"2024-02-06T21:43:08.871254+0000","last_unstale":"2024-02-06T22:17:52.368275+0000","last_undegraded":"2024-02-06T22:17:52.368275+0000","last_fullsized":"2024-02-06T22:17:52.368275+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:02:54.631582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":2582,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.384957+0000","last_change":"2024-02-06T21:42:55.060952+0000","last_active":"2024-02-06T22:17:53.384957+0000","last_peered":"2024-02-06T22:17:53.384957+0000","last_clean":"2024-02-06T22:17:53.384957+0000","last_became_active":"2024-02-06T21:42:55.034062+0000","last_became_peered":"2024-02-06T21:42:55.034062+0000","last_unstale":"2024-02-06T22:17:53.384957+0000","last_undegraded":"2024-02-06T22:17:53.384957+0000","last_fullsized":"2024-02-06T22:17:53.384957+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T07:52:53.348265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":2572,"reported_epoch":1341,"state":"active+clean","last_fresh":"2024-02-06T22:17:52.376138+0000","last_change":"2024-02-06T21:43:15.867032+0000","last_active":"2024-02-06T22:17:52.376138+0000","last_peered":"2024-02-06T22:17:52.376138+0000","last_clean":"2024-02-06T22:17:52.376138+0000","last_became_active":"2024-02-06T21:43:15.861140+0000","last_became_peered":"2024-02-06T21:43:15.861140+0000","last_unstale":"2024-02-06T22:17:52.376138+0000","last_undegraded":"2024-02-06T22:17:52.376138+0000","last_fullsized":"2024-02-06T22:17:52.376138+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:00:40.646612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":2568,"reported_epoch":1339,"state":"active+clean","last_fresh":"2024-02-06T22:17:50.529638+0000","last_change":"2024-02-06T21:42:55.175403+0000","last_active":"2024-02-06T22:17:50.529638+0000","last_peered":"2024-02-06T22:17:50.529638+0000","last_clean":"2024-02-06T22:17:50.529638+0000","last_became_active":"2024-02-06T21:42:55.168677+0000","last_became_peered":"2024-02-06T21:42:55.168677+0000","last_unstale":"2024-02-06T22:17:50.529638+0000","last_undegraded":"2024-02-06T22:17:50.529638+0000","last_fullsized":"2024-02-06T22:17:50.529638+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T07:31:47.782588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":2583,"reported_epoch":1340,"state":"active+clean","last_fresh":"2024-02-06T22:17:51.323256+0000","last_change":"2024-02-06T21:42:52.000053+0000","last_active":"2024-02-06T22:17:51.323256+0000","last_peered":"2024-02-06T22:17:51.323256+0000","last_clean":"2024-02-06T22:17:51.323256+0000","last_became_active":"2024-02-06T21:42:51.993812+0000","last_became_peered":"2024-02-06T21:42:51.993812+0000","last_unstale":"2024-02-06T22:17:51.323256+0000","last_undegraded":"2024-02-06T22:17:51.323256+0000","last_fullsized":"2024-02-06T22:17:51.323256+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:44:21.306346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":2572,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.382261+0000","last_change":"2024-02-06T21:44:01.005089+0000","last_active":"2024-02-06T22:17:53.382261+0000","last_peered":"2024-02-06T22:17:53.382261+0000","last_clean":"2024-02-06T22:17:53.382261+0000","last_became_active":"2024-02-06T21:44:00.992836+0000","last_became_peered":"2024-02-06T21:44:00.992836+0000","last_unstale":"2024-02-06T22:17:53.382261+0000","last_undegraded":"2024-02-06T22:17:53.382261+0000","last_fullsized":"2024-02-06T22:17:53.382261+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:34:43.433551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":2580,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.359066+0000","last_change":"2024-02-06T21:43:14.066712+0000","last_active":"2024-02-06T22:17:53.359066+0000","last_peered":"2024-02-06T22:17:53.359066+0000","last_clean":"2024-02-06T22:17:53.359066+0000","last_became_active":"2024-02-06T21:43:14.057258+0000","last_became_peered":"2024-02-06T21:43:14.057258+0000","last_unstale":"2024-02-06T22:17:53.359066+0000","last_undegraded":"2024-02-06T22:17:53.359066+0000","last_fullsized":"2024-02-06T22:17:53.359066+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T21:49:47.557758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":2582,"reported_epoch":1343,"state":"active+clean","last_fresh":"2024-02-06T22:17:54.416037+0000","last_change":"2024-02-06T21:42:49.763089+0000","last_active":"2024-02-06T22:17:54.416037+0000","last_peered":"2024-02-06T22:17:54.416037+0000","last_clean":"2024-02-06T22:17:54.416037+0000","last_became_active":"2024-02-06T21:42:49.754000+0000","last_became_peered":"2024-02-06T21:42:49.754000+0000","last_unstale":"2024-02-06T22:17:54.416037+0000","last_undegraded":"2024-02-06T22:17:54.416037+0000","last_fullsized":"2024-02-06T22:17:54.416037+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T06:23:56.755422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":18,"seq":77309411767,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72632,"kb_used_data":5352,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299208,"statfs":{"total":96636764160,"available":96562388992,"internally_reserved":0,"allocated":5480448,"data_stored":2758305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:41 2024","interfaces":[{"interface":"back","average":{"1min":15.664999999999999,"5min":29.364000000000001,"15min":24.5},"min":{"1min":5.9740000000000002,"5min":4.96,"15min":4.5209999999999999},"max":{"1min":76.638000000000005,"5min":223.274,"15min":549.32299999999998},"last":9.5389999999999997},{"interface":"front","average":{"1min":20.262,"5min":35.253999999999998,"15min":29.021999999999998},"min":{"1min":10.795999999999999,"5min":7.9969999999999999,"15min":5.4900000000000002},"max":{"1min":78.369,"5min":224.31700000000001,"15min":368.31200000000001},"last":16.876000000000001}]},{"osd":2,"last update":"Tue Feb 6 22:17:09 2024","interfaces":[{"interface":"back","average":{"1min":23.908999999999999,"5min":30.920999999999999,"15min":26.542999999999999},"min":{"1min":11.26,"5min":5.1520000000000001,"15min":5.1520000000000001},"max":{"1min":88.763000000000005,"5min":205.28800000000001,"15min":387.75299999999999},"last":11.986000000000001},{"interface":"front","average":{"1min":12.706,"5min":29.155000000000001,"15min":31.622},"min":{"1min":5.3460000000000001,"5min":5.3460000000000001,"15min":5.3460000000000001},"max":{"1min":43.438000000000002,"5min":220.31899999999999,"15min":442.25},"last":13.981999999999999}]},{"osd":3,"last update":"Tue Feb 6 22:16:51 2024","interfaces":[{"interface":"back","average":{"1min":12.071,"5min":24.050999999999998,"15min":22.372},"min":{"1min":5.3899999999999997,"5min":5.3899999999999997,"15min":5.3899999999999997},"max":{"1min":27.318000000000001,"5min":359.19299999999998,"15min":359.19299999999998},"last":10.957000000000001},{"interface":"front","average":{"1min":24.745999999999999,"5min":33.384999999999998,"15min":31.391999999999999},"min":{"1min":11.438000000000001,"5min":8.5500000000000007,"15min":7.8010000000000002},"max":{"1min":100.857,"5min":266.40600000000001,"15min":462.86200000000002},"last":18.096}]},{"osd":4,"last update":"Tue Feb 6 22:17:24 2024","interfaces":[{"interface":"back","average":{"1min":25.393999999999998,"5min":39.593000000000004,"15min":29.221},"min":{"1min":12.670999999999999,"5min":9.0559999999999992,"15min":5.9539999999999997},"max":{"1min":85.082999999999998,"5min":535.22500000000002,"15min":535.22500000000002},"last":20.530000000000001},{"interface":"front","average":{"1min":12.597,"5min":28.677,"15min":22.689},"min":{"1min":7.5220000000000002,"5min":6.3010000000000002,"15min":5.4649999999999999},"max":{"1min":49.466000000000001,"5min":473.69799999999998,"15min":473.69799999999998},"last":8.3940000000000001}]},{"osd":5,"last update":"Tue Feb 6 22:17:41 2024","interfaces":[{"interface":"back","average":{"1min":27.221,"5min":36.164999999999999,"15min":32.499000000000002},"min":{"1min":14.074999999999999,"5min":8.0459999999999994,"15min":6.4550000000000001},"max":{"1min":90.605999999999995,"5min":260.74400000000003,"15min":454.49099999999999},"last":21.324000000000002},{"interface":"front","average":{"1min":17.254000000000001,"5min":25.736999999999998,"15min":26.802},"min":{"1min":8.1069999999999993,"5min":6.8010000000000002,"15min":6.109},"max":{"1min":70.180000000000007,"5min":147.011,"15min":539.04700000000003},"last":14.577}]},{"osd":6,"last update":"Tue Feb 6 22:17:24 2024","interfaces":[{"interface":"back","average":{"1min":27.564,"5min":40.116999999999997,"15min":37.177},"min":{"1min":13.347,"5min":12.289,"15min":10.353999999999999},"max":{"1min":89.951999999999998,"5min":268.73399999999998,"15min":482.846},"last":22.105},{"interface":"front","average":{"1min":16.326000000000001,"5min":27.145,"15min":24.978000000000002},"min":{"1min":8.7759999999999998,"5min":6.9349999999999996,"15min":6.9349999999999996},"max":{"1min":51.412999999999997,"5min":186.95500000000001,"15min":322.64999999999998},"last":13.06}]},{"osd":7,"last update":"Tue Feb 6 22:17:41 2024","interfaces":[{"interface":"back","average":{"1min":15.381,"5min":27.943000000000001,"15min":30.135000000000002},"min":{"1min":6.1669999999999998,"5min":6.1669999999999998,"15min":6.1669999999999998},"max":{"1min":75.899000000000001,"5min":249.69200000000001,"15min":452.22399999999999},"last":6.3639999999999999},{"interface":"front","average":{"1min":31.131,"5min":42.412999999999997,"15min":35.392000000000003},"min":{"1min":13.923999999999999,"5min":10.964,"15min":7.7030000000000003},"max":{"1min":99.406000000000006,"5min":292.94600000000003,"15min":563.26900000000001},"last":19.073}]}]},{"osd":6,"up_from":17,"seq":73014444472,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72248,"kb_used_data":4972,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299592,"statfs":{"total":96636764160,"available":96562782208,"internally_reserved":0,"allocated":5091328,"data_stored":2367949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":14.135,"5min":21.481000000000002,"15min":17.268000000000001},"min":{"1min":4.3339999999999996,"5min":4.3230000000000004,"15min":4.3230000000000004},"max":{"1min":59.670999999999999,"5min":188.761,"15min":218.85400000000001},"last":13.481999999999999},{"interface":"front","average":{"1min":11.811,"5min":24.353000000000002,"15min":17.314},"min":{"1min":5.1559999999999997,"5min":5.1559999999999997,"15min":4.8449999999999998},"max":{"1min":38.082999999999998,"5min":193.68000000000001,"15min":193.68000000000001},"last":7.5709999999999997}]},{"osd":1,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":15.811999999999999,"5min":23.908999999999999,"15min":21.940000000000001},"min":{"1min":6.431,"5min":6.1459999999999999,"15min":5.109},"max":{"1min":54.000999999999998,"5min":180.447,"15min":204.64500000000001},"last":12.768000000000001},{"interface":"front","average":{"1min":17.199000000000002,"5min":23.283000000000001,"15min":24.012},"min":{"1min":6.7949999999999999,"5min":6.7949999999999999,"15min":6.1070000000000002},"max":{"1min":60.319000000000003,"5min":168.404,"15min":460.71600000000001},"last":8.9610000000000003}]},{"osd":2,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":19.161000000000001,"5min":36.042000000000002,"15min":26.562999999999999},"min":{"1min":8.0570000000000004,"5min":6.9359999999999999,"15min":5.5330000000000004},"max":{"1min":72.200000000000003,"5min":794.04200000000003,"15min":794.04200000000003},"last":11.016},{"interface":"front","average":{"1min":19.587,"5min":43.002000000000002,"15min":27.474},"min":{"1min":11.856,"5min":6.2999999999999998,"15min":5.601},"max":{"1min":62.237000000000002,"5min":795.17499999999995,"15min":795.17499999999995},"last":15.772}]},{"osd":3,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":19.741,"5min":32.057000000000002,"15min":24.117000000000001},"min":{"1min":10.901999999999999,"5min":7.718,"15min":5.9240000000000004},"max":{"1min":72.951999999999998,"5min":317.899,"15min":317.899},"last":11.653},{"interface":"front","average":{"1min":20.227,"5min":29.085999999999999,"15min":21.300999999999998},"min":{"1min":10.553000000000001,"5min":6.468,"15min":6.0380000000000003},"max":{"1min":65.974000000000004,"5min":405.108,"15min":405.108},"last":16.552}]},{"osd":4,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":10.769,"5min":23.568999999999999,"15min":22.443999999999999},"min":{"1min":5.8940000000000001,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":30.241,"5min":231.55699999999999,"15min":243.667},"last":6.4859999999999998},{"interface":"front","average":{"1min":14.613,"5min":30.960000000000001,"15min":27.713000000000001},"min":{"1min":8.641,"5min":7.7160000000000002,"15min":7.3399999999999999},"max":{"1min":39.484999999999999,"5min":284.37,"15min":339.34399999999999},"last":9.9689999999999994}]},{"osd":5,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":20.939,"5min":24.574999999999999,"15min":22.995000000000001},"min":{"1min":11.087999999999999,"5min":9.2929999999999993,"15min":7.5990000000000002},"max":{"1min":67.230000000000004,"5min":157.095,"15min":240.20500000000001},"last":14.972},{"interface":"front","average":{"1min":23.254999999999999,"5min":31.707999999999998,"15min":27.795000000000002},"min":{"1min":14.375,"5min":13.481,"15min":9.9260000000000002},"max":{"1min":69.085999999999999,"5min":202.673,"15min":223.42500000000001},"last":18.164999999999999}]},{"osd":7,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":23.655999999999999,"5min":30.402999999999999,"15min":23.09},"min":{"1min":15.859,"5min":11.272,"15min":7.4580000000000002},"max":{"1min":70.263999999999996,"5min":190.77799999999999,"15min":224.63},"last":17.344000000000001},{"interface":"front","average":{"1min":21.408999999999999,"5min":31.376999999999999,"15min":27.039999999999999},"min":{"1min":8.7430000000000003,"5min":8.7430000000000003,"15min":8.7430000000000003},"max":{"1min":74.259,"5min":203.26499999999999,"15min":220.893},"last":12.231}]}]},{"osd":5,"up_from":17,"seq":73014444474,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72604,"kb_used_data":5324,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299236,"statfs":{"total":96636764160,"available":96562417664,"internally_reserved":0,"allocated":5451776,"data_stored":2742327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":14.396000000000001,"5min":19.771999999999998,"15min":16.498999999999999},"min":{"1min":6.2709999999999999,"5min":5.6180000000000003,"15min":4.5090000000000003},"max":{"1min":72.215999999999994,"5min":115.977,"15min":350.07600000000002},"last":13.756},{"interface":"front","average":{"1min":20.567,"5min":26.533999999999999,"15min":22.433},"min":{"1min":9.1340000000000003,"5min":7.8810000000000002,"15min":6.133},"max":{"1min":81.966999999999999,"5min":128.446,"15min":351.94},"last":15.007999999999999}]},{"osd":1,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":16.960000000000001,"5min":22.443000000000001,"15min":21.648},"min":{"1min":7.9630000000000001,"5min":7.0830000000000002,"15min":6.2359999999999998},"max":{"1min":46.289000000000001,"5min":158.76400000000001,"15min":275.12},"last":14.387},{"interface":"front","average":{"1min":19.393000000000001,"5min":24.099,"15min":22.812999999999999},"min":{"1min":8.5939999999999994,"5min":8.1590000000000007,"15min":4.6520000000000001},"max":{"1min":61.463000000000001,"5min":112.78700000000001,"15min":369.73200000000003},"last":15.612}]},{"osd":2,"last update":"Tue Feb 6 22:17:21 2024","interfaces":[{"interface":"back","average":{"1min":12.752000000000001,"5min":43.362000000000002,"15min":31.550000000000001},"min":{"1min":5.7140000000000004,"5min":5.7140000000000004,"15min":5.7140000000000004},"max":{"1min":46.295000000000002,"5min":2330.5610000000001,"15min":2330.5610000000001},"last":7.3019999999999996},{"interface":"front","average":{"1min":17.791,"5min":50.844999999999999,"15min":35.262999999999998},"min":{"1min":7.3230000000000004,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":44.875999999999998,"5min":2614.4409999999998,"15min":2614.4409999999998},"last":19.096}]},{"osd":3,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":18.696000000000002,"5min":23.902000000000001,"15min":24.218},"min":{"1min":7.8049999999999997,"5min":6.125,"15min":6.125},"max":{"1min":62.171999999999997,"5min":156.80000000000001,"15min":393.55200000000002},"last":10.568},{"interface":"front","average":{"1min":14.869,"5min":19.66,"15min":21.309999999999999},"min":{"1min":6.1159999999999997,"5min":5.3819999999999997,"15min":5.3819999999999997},"max":{"1min":63.813000000000002,"5min":104.755,"15min":376.113},"last":8.3789999999999996}]},{"osd":4,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":21.885000000000002,"15min":23.100999999999999},"min":{"1min":6.5650000000000004,"5min":6.093,"15min":6.093},"max":{"1min":84.843999999999994,"5min":105.532,"15min":426.77300000000002},"last":13.170999999999999},{"interface":"front","average":{"1min":18.773,"5min":21.994,"15min":24.087},"min":{"1min":7.4870000000000001,"5min":7.3819999999999997,"15min":5.8330000000000002},"max":{"1min":50.149000000000001,"5min":90.284000000000006,"15min":419.16000000000003},"last":17.120999999999999}]},{"osd":6,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":20.283000000000001,"5min":29.465,"15min":28.431999999999999},"min":{"1min":6.9560000000000004,"5min":6.9560000000000004,"15min":6.9560000000000004},"max":{"1min":85.828000000000003,"5min":219.14099999999999,"15min":392.71600000000001},"last":12.541},{"interface":"front","average":{"1min":21.872,"5min":25.550000000000001,"15min":26.341000000000001},"min":{"1min":9.8740000000000006,"5min":7.2350000000000003,"15min":7.2350000000000003},"max":{"1min":65.367000000000004,"5min":152.69900000000001,"15min":377.31400000000002},"last":11.234999999999999}]},{"osd":7,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":19.959,"5min":25.904,"15min":25.827999999999999},"min":{"1min":9.0510000000000002,"5min":9.0510000000000002,"15min":9.0510000000000002},"max":{"1min":63.646000000000001,"5min":124.711,"15min":377.87400000000002},"last":16.288},{"interface":"front","average":{"1min":19.725000000000001,"5min":27.969999999999999,"15min":24.045999999999999},"min":{"1min":7.8940000000000001,"5min":7.8940000000000001,"15min":6.7549999999999999},"max":{"1min":48.325000000000003,"5min":152.59399999999999,"15min":338.803},"last":9.9830000000000005}]}]},{"osd":4,"up_from":17,"seq":73014444471,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72244,"kb_used_data":4968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299596,"statfs":{"total":96636764160,"available":96562786304,"internally_reserved":0,"allocated":5087232,"data_stored":2367944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.2439999999999998,"5min":17.853000000000002,"15min":16.888999999999999},"min":{"1min":4.907,"5min":4.4169999999999998,"15min":4.3899999999999997},"max":{"1min":19.225999999999999,"5min":167.291,"15min":216.86600000000001},"last":7.6379999999999999},{"interface":"front","average":{"1min":9.9450000000000003,"5min":22.707999999999998,"15min":18.631},"min":{"1min":4.742,"5min":4.6920000000000002,"15min":4.6829999999999998},"max":{"1min":23.745000000000001,"5min":231.82599999999999,"15min":287.97300000000001},"last":14.872}]},{"osd":1,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":17.431999999999999,"5min":24.254999999999999,"15min":30.890000000000001},"min":{"1min":6.2949999999999999,"5min":6.2949999999999999,"15min":5.1539999999999999},"max":{"1min":65.745999999999995,"5min":207.22399999999999,"15min":678.16999999999996},"last":9.7309999999999999},{"interface":"front","average":{"1min":15.343999999999999,"5min":23.465,"15min":32.118000000000002},"min":{"1min":8.4299999999999997,"5min":6.2599999999999998,"15min":6.2599999999999998},"max":{"1min":25.268999999999998,"5min":203.19999999999999,"15min":667.16300000000001},"last":11.487}]},{"osd":2,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":15.932,"5min":59.622999999999998,"15min":35.741999999999997},"min":{"1min":10.554,"5min":6.601,"15min":6.601},"max":{"1min":32.744999999999997,"5min":2277.7020000000002,"15min":2277.7020000000002},"last":15.648999999999999},{"interface":"front","average":{"1min":22.949999999999999,"5min":66.596999999999994,"15min":44.031999999999996},"min":{"1min":12.722,"5min":9.4559999999999995,"15min":9.4559999999999995},"max":{"1min":52.287999999999997,"5min":2279.0419999999999,"15min":2279.0419999999999},"last":17.969000000000001}]},{"osd":3,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":21.295999999999999,"5min":30.468,"15min":28.460999999999999},"min":{"1min":9.4499999999999993,"5min":6.3559999999999999,"15min":6.3559999999999999},"max":{"1min":62.805999999999997,"5min":254.292,"15min":423.20400000000001},"last":17.077999999999999},{"interface":"front","average":{"1min":15.101000000000001,"5min":31.655000000000001,"15min":24.419},"min":{"1min":6.9610000000000003,"5min":6.9610000000000003,"15min":5.5910000000000002},"max":{"1min":25.286999999999999,"5min":225.64699999999999,"15min":359.959},"last":12.189}]},{"osd":5,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":22.587,"5min":34.43,"15min":29.388999999999999},"min":{"1min":12.337999999999999,"5min":10.756,"15min":7.2999999999999998},"max":{"1min":66.715999999999994,"5min":248.70500000000001,"15min":363.36900000000003},"last":20.989000000000001},{"interface":"front","average":{"1min":15.039999999999999,"5min":28.181999999999999,"15min":27.018999999999998},"min":{"1min":6.6040000000000001,"5min":6.476,"15min":6.1550000000000002},"max":{"1min":43.223999999999997,"5min":208.02699999999999,"15min":809.58100000000002},"last":10.851000000000001}]},{"osd":6,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":17.777999999999999,"5min":24.445,"15min":26.818999999999999},"min":{"1min":7.5819999999999999,"5min":6.2930000000000001,"15min":6.2930000000000001},"max":{"1min":63.929000000000002,"5min":174.40700000000001,"15min":400.46800000000002},"last":12.803000000000001},{"interface":"front","average":{"1min":24.074999999999999,"5min":35.155000000000001,"15min":36.561},"min":{"1min":13.917,"5min":11.411,"15min":10.502000000000001},"max":{"1min":66.519999999999996,"5min":269.44200000000001,"15min":498.22300000000001},"last":20.058}]},{"osd":7,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":24.492000000000001,"5min":35.420000000000002,"15min":26.943000000000001},"min":{"1min":14.81,"5min":8.9009999999999998,"15min":6.6239999999999997},"max":{"1min":52.576999999999998,"5min":287.61599999999999,"15min":360.55599999999998},"last":18.937999999999999},{"interface":"front","average":{"1min":15.82,"5min":26.690999999999999,"15min":28.559999999999999},"min":{"1min":9.7210000000000001,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":28.702999999999999,"5min":215.26400000000001,"15min":478.649},"last":8.8460000000000001}]}]},{"osd":3,"up_from":17,"seq":73014444471,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72244,"kb_used_data":4968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299596,"statfs":{"total":96636764160,"available":96562786304,"internally_reserved":0,"allocated":5087232,"data_stored":2367944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:35 2024","interfaces":[{"interface":"back","average":{"1min":15.148999999999999,"5min":19.969999999999999,"15min":18.234000000000002},"min":{"1min":5.6100000000000003,"5min":4.5270000000000001,"15min":4.173},"max":{"1min":37.642000000000003,"5min":178.14400000000001,"15min":233.84200000000001},"last":10.285},{"interface":"front","average":{"1min":17.673999999999999,"5min":22.867000000000001,"15min":22.109999999999999},"min":{"1min":7.1299999999999999,"5min":5.3949999999999996,"15min":5.3949999999999996},"max":{"1min":45.851999999999997,"5min":224.822,"15min":224.822},"last":11.699999999999999}]},{"osd":1,"last update":"Tue Feb 6 22:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.183999999999999,"5min":19.210000000000001,"15min":21.574000000000002},"min":{"1min":6.7039999999999997,"5min":6.7039999999999997,"15min":6.7039999999999997},"max":{"1min":21.882999999999999,"5min":169.31100000000001,"15min":273.12700000000001},"last":14.042999999999999},{"interface":"front","average":{"1min":16.084,"5min":25.670000000000002,"15min":25.431999999999999},"min":{"1min":10.845000000000001,"5min":10.845000000000001,"15min":9.2880000000000003},"max":{"1min":24.265000000000001,"5min":205.495,"15min":322.70499999999998},"last":12.353}]},{"osd":2,"last update":"Tue Feb 6 22:17:35 2024","interfaces":[{"interface":"back","average":{"1min":25.116,"5min":28.814,"15min":26.242999999999999},"min":{"1min":11.129,"5min":11.129,"15min":9.1999999999999993},"max":{"1min":59.799999999999997,"5min":240.316,"15min":282.40699999999998},"last":13.276},{"interface":"front","average":{"1min":11.965,"5min":28.228999999999999,"15min":21.456},"min":{"1min":5.8979999999999997,"5min":5.8419999999999996,"15min":5.1550000000000002},"max":{"1min":39.423000000000002,"5min":998.23500000000001,"15min":998.23500000000001},"last":6.3099999999999996}]},{"osd":4,"last update":"Tue Feb 6 22:17:17 2024","interfaces":[{"interface":"back","average":{"1min":12.571999999999999,"5min":17.777000000000001,"15min":20.849},"min":{"1min":7.6459999999999999,"5min":6.1109999999999998,"15min":5.8339999999999996},"max":{"1min":25.420000000000002,"5min":162.65299999999999,"15min":189.12},"last":8.266},{"interface":"front","average":{"1min":18.823,"5min":27.978999999999999,"15min":31.401},"min":{"1min":12.393000000000001,"5min":11.710000000000001,"15min":11.045},"max":{"1min":30.169,"5min":176.35400000000001,"15min":235.315},"last":15.599}]},{"osd":5,"last update":"Tue Feb 6 22:17:35 2024","interfaces":[{"interface":"back","average":{"1min":29.649000000000001,"5min":32.82,"15min":29.718},"min":{"1min":10.526,"5min":10.087999999999999,"15min":8.9920000000000009},"max":{"1min":77.253,"5min":264.06999999999999,"15min":626.27700000000004},"last":14.845000000000001},{"interface":"front","average":{"1min":37.348999999999997,"5min":37.712000000000003,"15min":32.811999999999998},"min":{"1min":13.699,"5min":13.699,"15min":13.699},"max":{"1min":112.84699999999999,"5min":284.88299999999998,"15min":614.85400000000004},"last":16.399000000000001}]},{"osd":6,"last update":"Tue Feb 6 22:16:52 2024","interfaces":[{"interface":"back","average":{"1min":22.713999999999999,"5min":24.713999999999999,"15min":26.106999999999999},"min":{"1min":8.7240000000000002,"5min":6.4550000000000001,"15min":6.218},"max":{"1min":55.887,"5min":177.40700000000001,"15min":954.18200000000002},"last":17.332000000000001},{"interface":"front","average":{"1min":16.166,"5min":29.777000000000001,"15min":35.177},"min":{"1min":8.3729999999999993,"5min":8.3729999999999993,"15min":8.3729999999999993},"max":{"1min":40.037999999999997,"5min":209.25800000000001,"15min":954.79300000000001},"last":7.5289999999999999}]},{"osd":7,"last update":"Tue Feb 6 22:16:52 2024","interfaces":[{"interface":"back","average":{"1min":15.278,"5min":21.925000000000001,"15min":25.556999999999999},"min":{"1min":8.2970000000000006,"5min":7.3579999999999997,"15min":7.319},"max":{"1min":40.340000000000003,"5min":173.642,"15min":655.048},"last":9.3759999999999994},{"interface":"front","average":{"1min":27.821999999999999,"5min":31.497,"15min":33.704999999999998},"min":{"1min":16.033000000000001,"5min":14.802,"15min":6.9199999999999999},"max":{"1min":69.661000000000001,"5min":206.87,"15min":653.67600000000004},"last":18.169}]}]},{"osd":2,"up_from":17,"seq":73014444471,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72244,"kb_used_data":4968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299596,"statfs":{"total":96636764160,"available":96562786304,"internally_reserved":0,"allocated":5087232,"data_stored":2367944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":9.8650000000000002,"5min":40.578000000000003,"15min":27.196999999999999},"min":{"1min":5.3460000000000001,"5min":4.6050000000000004,"15min":4.3150000000000004},"max":{"1min":23.093,"5min":1631.963,"15min":1631.963},"last":13.194000000000001},{"interface":"front","average":{"1min":9.0060000000000002,"5min":21.448,"15min":21.940999999999999},"min":{"1min":4.4859999999999998,"5min":4.4859999999999998,"15min":3.9630000000000001},"max":{"1min":17.125,"5min":138.09800000000001,"15min":370.68799999999999},"last":8.4320000000000004}]},{"osd":1,"last update":"Tue Feb 6 22:17:26 2024","interfaces":[{"interface":"back","average":{"1min":10.836,"5min":52.090000000000003,"15min":32.5},"min":{"1min":6.9000000000000004,"5min":5.9340000000000002,"15min":4.492},"max":{"1min":38.810000000000002,"5min":3146.2530000000002,"15min":3146.2530000000002},"last":6.0919999999999996},{"interface":"front","average":{"1min":19.111000000000001,"5min":58.341999999999999,"15min":36.834000000000003},"min":{"1min":8.4320000000000004,"5min":6.9669999999999996,"15min":6.9020000000000001},"max":{"1min":111.504,"5min":3147.6979999999999,"15min":3147.6979999999999},"last":9.0259999999999998}]},{"osd":3,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":11.875999999999999,"5min":51.591999999999999,"15min":33.267000000000003},"min":{"1min":8.4879999999999995,"5min":7.3890000000000002,"15min":6.1239999999999997},"max":{"1min":19.702999999999999,"5min":2357.8649999999998,"15min":2357.8649999999998},"last":7.7939999999999996},{"interface":"front","average":{"1min":17.119,"5min":58.719000000000001,"15min":37.293999999999997},"min":{"1min":9.6289999999999996,"5min":9.6289999999999996,"15min":7.133},"max":{"1min":27.391999999999999,"5min":2367.828,"15min":2367.828},"last":15.157}]},{"osd":4,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":12.506,"5min":45.119,"15min":36.264000000000003},"min":{"1min":7.5949999999999998,"5min":6.5129999999999999,"15min":6.5129999999999999},"max":{"1min":17.905000000000001,"5min":1805.6099999999999,"15min":1805.6099999999999},"last":10.436999999999999},{"interface":"front","average":{"1min":15.993,"5min":22.584,"15min":26.597999999999999},"min":{"1min":10.06,"5min":6.976,"15min":6.4009999999999998},"max":{"1min":26.065999999999999,"5min":139.29900000000001,"15min":366.25099999999998},"last":11.781000000000001}]},{"osd":5,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":13.83,"5min":59.646000000000001,"15min":39.192},"min":{"1min":8.7680000000000007,"5min":7.0650000000000004,"15min":6.9610000000000003},"max":{"1min":21.802,"5min":3145.4110000000001,"15min":3145.4110000000001},"last":11.118},{"interface":"front","average":{"1min":18.478000000000002,"5min":58.215000000000003,"15min":41.988999999999997},"min":{"1min":12.645,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":28.748000000000001,"5min":2302.047,"15min":2302.047},"last":14.541}]},{"osd":6,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":13.406000000000001,"5min":31.114000000000001,"15min":30.577999999999999},"min":{"1min":10.297000000000001,"5min":7.0819999999999999,"15min":6.6479999999999997},"max":{"1min":19.422999999999998,"5min":266.53500000000003,"15min":369.185},"last":12.52},{"interface":"front","average":{"1min":19.795000000000002,"5min":60.094000000000001,"15min":42.265999999999998},"min":{"1min":14.802,"5min":9.4570000000000007,"15min":8.4930000000000003},"max":{"1min":30.856000000000002,"5min":2302.8209999999999,"15min":2302.8209999999999},"last":17.099}]},{"osd":7,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":15.435,"5min":27.277000000000001,"15min":28.007999999999999},"min":{"1min":9.0050000000000008,"5min":8.5790000000000006,"15min":8.4619999999999997},"max":{"1min":23.422000000000001,"5min":267.28300000000002,"15min":405.29899999999998},"last":13.826000000000001},{"interface":"front","average":{"1min":17.768999999999998,"5min":31.587,"15min":31.151},"min":{"1min":12.055,"5min":11.515000000000001,"15min":10.06},"max":{"1min":25.113,"5min":268.60899999999998,"15min":342.22699999999998},"last":18.021999999999998}]}]},{"osd":7,"up_from":13,"seq":55834575286,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76368,"kb_used_data":4932,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295472,"statfs":{"total":96636764160,"available":96558563328,"internally_reserved":0,"allocated":5050368,"data_stored":2344487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":11.686999999999999,"5min":23.573,"15min":18.436},"min":{"1min":5.0789999999999997,"5min":4.2880000000000003,"15min":4.2880000000000003},"max":{"1min":28.937000000000001,"5min":283.42899999999997,"15min":283.42899999999997},"last":19.643999999999998},{"interface":"front","average":{"1min":12.000999999999999,"5min":23.219999999999999,"15min":19.027999999999999},"min":{"1min":5.2949999999999999,"5min":4.3730000000000002,"15min":4.3730000000000002},"max":{"1min":33.189,"5min":279.459,"15min":387.32999999999998},"last":7.335}]},{"osd":1,"last update":"Tue Feb 6 22:17:29 2024","interfaces":[{"interface":"back","average":{"1min":13.608000000000001,"5min":19.052,"15min":22.741},"min":{"1min":6.0609999999999999,"5min":6.0609999999999999,"15min":5.577},"max":{"1min":31.280999999999999,"5min":145.70599999999999,"15min":325.41699999999997},"last":11.724},{"interface":"front","average":{"1min":13.199999999999999,"5min":17.952000000000002,"15min":21.225999999999999},"min":{"1min":6.2720000000000002,"5min":6.2720000000000002,"15min":4.7960000000000003},"max":{"1min":30.079999999999998,"5min":159.56100000000001,"15min":380.70800000000003},"last":18.518000000000001}]},{"osd":2,"last update":"Tue Feb 6 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":15.321999999999999,"5min":78.852000000000004,"15min":42.732999999999997},"min":{"1min":7.1890000000000001,"5min":6.367,"15min":5.8769999999999998},"max":{"1min":56.795000000000002,"5min":3537.9119999999998,"15min":3537.9119999999998},"last":9.8010000000000002},{"interface":"front","average":{"1min":17,"5min":81.25,"15min":44.497},"min":{"1min":9.3260000000000005,"5min":7.6790000000000003,"15min":6.6539999999999999},"max":{"1min":26.536999999999999,"5min":3538.9259999999999,"15min":3538.9259999999999},"last":21.727}]},{"osd":3,"last update":"Tue Feb 6 22:17:29 2024","interfaces":[{"interface":"back","average":{"1min":16.347999999999999,"5min":20.353999999999999,"15min":21.109999999999999},"min":{"1min":7.7990000000000004,"5min":6.8399999999999999,"15min":6.3390000000000004},"max":{"1min":57.917999999999999,"5min":186.297,"15min":317.94600000000003},"last":15.587999999999999},{"interface":"front","average":{"1min":22.106000000000002,"5min":27.684000000000001,"15min":27.027999999999999},"min":{"1min":14.65,"5min":9.8360000000000003,"15min":9.8360000000000003},"max":{"1min":58.795000000000002,"5min":201.75299999999999,"15min":325.60599999999999},"last":22.898}]},{"osd":4,"last update":"Tue Feb 6 22:17:20 2024","interfaces":[{"interface":"back","average":{"1min":20.239999999999998,"5min":22.088999999999999,"15min":22.254999999999999},"min":{"1min":9.7159999999999993,"5min":9.2370000000000001,"15min":6.9690000000000003},"max":{"1min":61.972999999999999,"5min":102.001,"15min":235.27799999999999},"last":10.444000000000001},{"interface":"front","average":{"1min":14.773999999999999,"5min":18.849,"15min":25.335000000000001},"min":{"1min":7.585,"5min":6.8440000000000003,"15min":6.8440000000000003},"max":{"1min":25.353000000000002,"5min":87.968999999999994,"15min":350.33300000000003},"last":14.77}]},{"osd":5,"last update":"Tue Feb 6 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":24.497,"5min":25.561,"15min":24.407},"min":{"1min":12.058,"5min":7.1189999999999998,"15min":7.1189999999999998},"max":{"1min":89.010999999999996,"5min":201.70400000000001,"15min":214.83500000000001},"last":20.087},{"interface":"front","average":{"1min":18.481999999999999,"5min":22.195,"15min":25.890999999999998},"min":{"1min":8.6969999999999992,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":65.813999999999993,"5min":106.07299999999999,"15min":301.78399999999999},"last":24.082999999999998}]},{"osd":6,"last update":"Tue Feb 6 22:17:29 2024","interfaces":[{"interface":"back","average":{"1min":23.632999999999999,"5min":26.312999999999999,"15min":25.606000000000002},"min":{"1min":15.241,"5min":8.3239999999999998,"15min":8.3239999999999998},"max":{"1min":60.616,"5min":203.58199999999999,"15min":235.27799999999999},"last":25.675999999999998},{"interface":"front","average":{"1min":16.300999999999998,"5min":22.77,"15min":26.331},"min":{"1min":9.3000000000000007,"5min":7.3179999999999996,"15min":6.7400000000000002},"max":{"1min":28.381,"5min":185.49799999999999,"15min":490.68299999999999},"last":13.365}]}]},{"osd":0,"up_from":13,"seq":55834575283,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72208,"kb_used_data":4932,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299632,"statfs":{"total":96636764160,"available":96562823168,"internally_reserved":0,"allocated":5050368,"data_stored":2344487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 6 22:16:50 2024","interfaces":[{"interface":"back","average":{"1min":15.423,"5min":27.477,"15min":24.606000000000002},"min":{"1min":5.4459999999999997,"5min":4.5460000000000003,"15min":4.4029999999999996},"max":{"1min":93.102999999999994,"5min":314.11000000000001,"15min":416.55599999999998},"last":6.6349999999999998},{"interface":"front","average":{"1min":11.308999999999999,"5min":21.106000000000002,"15min":23.184999999999999},"min":{"1min":5.8769999999999998,"5min":5.5389999999999997,"15min":4.5},"max":{"1min":20.073,"5min":199.238,"15min":407.44799999999998},"last":9.0109999999999992}]},{"osd":2,"last update":"Tue Feb 6 22:16:50 2024","interfaces":[{"interface":"back","average":{"1min":18.163,"5min":27.111999999999998,"15min":29.199000000000002},"min":{"1min":6.4880000000000004,"5min":6.4880000000000004,"15min":6.3799999999999999},"max":{"1min":72.644000000000005,"5min":264.48200000000003,"15min":395.14800000000002},"last":12.398999999999999},{"interface":"front","average":{"1min":15.352,"5min":27.988,"15min":25.997},"min":{"1min":5.8390000000000004,"5min":5.5259999999999998,"15min":5.383},"max":{"1min":62.317,"5min":514.94500000000005,"15min":514.94500000000005},"last":11.593999999999999}]},{"osd":3,"last update":"Tue Feb 6 22:16:50 2024","interfaces":[{"interface":"back","average":{"1min":14.545,"5min":24.606999999999999,"15min":25.003},"min":{"1min":7.633,"5min":6.351,"15min":5.8140000000000001},"max":{"1min":26.542999999999999,"5min":291.59100000000001,"15min":376.95100000000002},"last":27.548999999999999},{"interface":"front","average":{"1min":14.034000000000001,"5min":25.123000000000001,"15min":26.632999999999999},"min":{"1min":6.6139999999999999,"5min":6.6139999999999999,"15min":6.6139999999999999},"max":{"1min":26.699999999999999,"5min":262.06900000000002,"15min":395.83100000000002},"last":7.75}]},{"osd":4,"last update":"Tue Feb 6 22:17:09 2024","interfaces":[{"interface":"back","average":{"1min":16.298999999999999,"5min":31.213000000000001,"15min":28.899000000000001},"min":{"1min":9.8650000000000002,"5min":7.3609999999999998,"15min":6.6970000000000001},"max":{"1min":27.573,"5min":491.53100000000001,"15min":491.53100000000001},"last":14.371},{"interface":"front","average":{"1min":11.919,"5min":26.989999999999998,"15min":25.725000000000001},"min":{"1min":8.4060000000000006,"5min":7.2409999999999997,"15min":6.6589999999999998},"max":{"1min":18.683,"5min":490.28699999999998,"15min":490.28699999999998},"last":10.390000000000001}]},{"osd":5,"last update":"Tue Feb 6 22:16:47 2024","interfaces":[{"interface":"back","average":{"1min":21.765000000000001,"5min":30.367999999999999,"15min":29.795000000000002},"min":{"1min":9.1989999999999998,"5min":7.9969999999999999,"15min":7.9969999999999999},"max":{"1min":41.332000000000001,"5min":282.00799999999998,"15min":480.24200000000002},"last":10.941000000000001},{"interface":"front","average":{"1min":16.699000000000002,"5min":25.358000000000001,"15min":25.074999999999999},"min":{"1min":9.0739999999999998,"5min":8.9350000000000005,"15min":6.8620000000000001},"max":{"1min":27.058,"5min":193.21700000000001,"15min":347.78699999999998},"last":9.7750000000000004}]},{"osd":6,"last update":"Tue Feb 6 22:16:47 2024","interfaces":[{"interface":"back","average":{"1min":25.838999999999999,"5min":31.405999999999999,"15min":31.593},"min":{"1min":11.106999999999999,"5min":9.4789999999999992,"15min":8.9139999999999997},"max":{"1min":85.287000000000006,"5min":218.12,"15min":550.89700000000005},"last":28.219999999999999},{"interface":"front","average":{"1min":19.417999999999999,"5min":26.103000000000002,"15min":26.722999999999999},"min":{"1min":9.5619999999999994,"5min":9.1310000000000002,"15min":8.3390000000000004},"max":{"1min":29.532,"5min":175.947,"15min":486.25299999999999},"last":22.055}]},{"osd":7,"last update":"Tue Feb 6 22:17:38 2024","interfaces":[{"interface":"back","average":{"1min":24.734000000000002,"5min":33.390999999999998,"15min":30.097000000000001},"min":{"1min":13.772,"5min":11.374000000000001,"15min":9.6170000000000009},"max":{"1min":107.01000000000001,"5min":218.108,"15min":404.22800000000001},"last":22.709},{"interface":"front","average":{"1min":28.641999999999999,"5min":38.777999999999999,"15min":34.070999999999998},"min":{"1min":15.483000000000001,"5min":13.800000000000001,"15min":11.497999999999999},"max":{"1min":109.504,"5min":219.56299999999999,"15min":415.92000000000002},"last":24.742999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-06T22:17:58.230 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-02-06T22:17:58.230 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-06T22:17:58.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.331+0000 7fe9c6de6640 10 monclient: get_monmap_and_config 2024-02-06T22:17:58.333 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.331+0000 7fe9c6de6640 10 monclient: build_initial_monmap 2024-02-06T22:17:58.333 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.331+0000 7fe9c6de6640 10 monclient: monmap: 2024-02-06T22:17:58.333 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:17:58.333 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:58.333 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:17:58.332718+0000 2024-02-06T22:17:58.333 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:17:58.332718+0000 2024-02-06T22:17:58.333 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:17:58.333 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:58.333 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:17:58.333 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:17:58.333 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:17:58.334 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:17:58.334 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:17:58.334 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:58.336 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.332+0000 7fe9c6de6640 1 Processor -- start 2024-02-06T22:17:58.336 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.332+0000 7fe9c6de6640 1 -- start start 2024-02-06T22:17:58.336 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.332+0000 7fe9c6de6640 10 monclient: init 2024-02-06T22:17:58.336 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.332+0000 7fe9c6de6640 10 monclient: _reopen_session rank -1 2024-02-06T22:17:58.336 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.332+0000 7fe9c6de6640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-02-06T22:17:58.336 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.332+0000 7fe9c6de6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9c00b54b0 addr v1:172.21.15.120:6789/0 2024-02-06T22:17:58.336 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.332+0000 7fe9c6de6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9c0140050 0x7fe9c00a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:58.336 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.332+0000 7fe9c6de6640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c0140050 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:17:58.336 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.332+0000 7fe9c6de6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9c0142680 addr v1:172.21.15.135:6789/0 2024-02-06T22:17:58.336 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.332+0000 7fe9c6de6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c0143220 con 0x7fe9c00b54b0 2024-02-06T22:17:58.336 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.332+0000 7fe9c6de6640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c00a9fc0 con 0x7fe9c0142680 2024-02-06T22:17:58.336 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.332+0000 7fe9c6de6640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:58.337 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.332+0000 7fe9c6de6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c00ab520 con 0x7fe9c0140050 2024-02-06T22:17:58.337 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.332+0000 7fe9c6de6640 10 monclient(hunting): _renew_subs 2024-02-06T22:17:58.337 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.332+0000 7fe9c6de6640 10 monclient(hunting): authenticate will time out at 3508.670325s 2024-02-06T22:17:58.337 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.333+0000 7fe9c4b82640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fe9c00b54b0 0x7fe9c013f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:50042/0 (socket says 172.21.15.120:50042) 2024-02-06T22:17:58.337 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.333+0000 7fe9c4b82640 1 -- 172.21.15.120:0/1982384033 learned_addr learned my addr 172.21.15.120:0/1982384033 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T22:17:58.337 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.333+0000 7fe9bffff640 1 --2- 172.21.15.120:0/1982384033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9c0140050 0x7fe9c00a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:58.337 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.333+0000 7fe9bffff640 10 monclient(hunting): get_auth_request con 0x7fe9c0140050 auth_method 0 2024-02-06T22:17:58.337 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.333+0000 7fe9bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:58.337 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.333+0000 7fe9bffff640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:58.337 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.333+0000 7fe9bffff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:58.337 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.334+0000 7fe9bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:58.337 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.334+0000 7fe9bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:58.337 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.334+0000 7fe9bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:58.338 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.335+0000 7fe9bf7fe640 1 -- 172.21.15.120:0/1982384033 <== mon.0 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107120561 0 0) 0x7fe9c0143220 con 0x7fe9c00b54b0 2024-02-06T22:17:58.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bf7fe640 10 monclient(hunting): _negotiate 2024-02-06T22:17:58.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bf7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:58.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:58.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bf7fe640 10 monclient(hunting): my global_id is 16494 2024-02-06T22:17:58.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bf7fe640 1 -- 172.21.15.120:0/1982384033 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b0003520 con 0x7fe9c00b54b0 2024-02-06T22:17:58.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bf7fe640 1 -- 172.21.15.120:0/1982384033 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444314441 0 0) 0x7fe9c00a9fc0 con 0x7fe9c0142680 2024-02-06T22:17:58.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bf7fe640 10 monclient(hunting): _negotiate 2024-02-06T22:17:58.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bf7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:58.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:58.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bf7fe640 10 monclient(hunting): my global_id is 21016 2024-02-06T22:17:58.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bf7fe640 1 -- 172.21.15.120:0/1982384033 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c0143220 con 0x7fe9c0142680 2024-02-06T22:17:58.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bffff640 10 monclient(hunting): handle_auth_done global_id 18680 payload 274 2024-02-06T22:17:58.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bffff640 1 -- 172.21.15.120:0/1982384033 >> v1:172.21.15.135:6789/0 conn(0x7fe9c0142680 legacy=0x7fe9c0142b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:58.340 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bffff640 1 -- 172.21.15.120:0/1982384033 >> v1:172.21.15.120:6789/0 conn(0x7fe9c00b54b0 legacy=0x7fe9c013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:58.340 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bffff640 10 monclient: _finish_hunting 0 2024-02-06T22:17:58.340 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bffff640 1 monclient: found mon.noname-b 2024-02-06T22:17:58.340 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:17:58.340 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T22:17:58.340 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.336+0000 7fe9bffff640 1 -- 172.21.15.120:0/1982384033 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c00ab720 con 0x7fe9c0140050 2024-02-06T22:17:58.340 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.338+0000 7fe9bffff640 1 --2- 172.21.15.120:0/1982384033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9c0140050 0x7fe9c00a9860 secure :-1 s=READY pgs=3339 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0097d0 tx=0x7fe9ac02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1e4e5abadb47d73 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:17:58.344 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9bf7fe640 1 -- 172.21.15.120:0/1982384033 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac031770 con 0x7fe9c0140050 2024-02-06T22:17:58.344 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:58.344 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9bf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:17:58.344 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9bf7fe640 10 monclient: dump: 2024-02-06T22:17:58.344 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:58.344 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:58.344 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:58.345 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:58.345 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:58.345 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:58.345 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:58.345 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:58.345 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:58.345 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:58.345 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9bf7fe640 10 monclient: _finish_auth 0 2024-02-06T22:17:58.345 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9bf7fe640 10 monclient: _check_auth_tickets 2024-02-06T22:17:58.345 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:17:58.345 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9bf7fe640 1 -- 172.21.15.120:0/1982384033 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ac0318d0 con 0x7fe9c0140050 2024-02-06T22:17:58.345 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9c6de6640 5 monclient: authenticate success, global_id 18680 2024-02-06T22:17:58.345 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9c6de6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T22:17:58.346 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:17:58.346 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9c6de6640 10 monclient: get_monmap_and_config success 2024-02-06T22:17:58.346 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9c6de6640 4 set_mon_vals no callback set 2024-02-06T22:17:58.346 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9bf7fe640 1 -- 172.21.15.120:0/1982384033 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac031c60 con 0x7fe9c0140050 2024-02-06T22:17:58.346 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:58.346 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T22:17:58.346 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.342+0000 7fe9bf7fe640 10 monclient: dump: 2024-02-06T22:17:58.346 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:58.346 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:58.346 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:58.346 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:58.346 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:58.346 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:58.346 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:58.346 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:58.347 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:58.347 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:58.347 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.343+0000 7fe9c6de6640 10 monclient: shutdown 2024-02-06T22:17:58.347 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.343+0000 7fe9c6de6640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:17:58.347 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.343+0000 7fe9c6de6640 1 -- 172.21.15.120:0/1982384033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9c0140050 msgr2=0x7fe9c00a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:58.347 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.343+0000 7fe9c6de6640 1 --2- 172.21.15.120:0/1982384033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9c0140050 0x7fe9c00a9860 secure :-1 s=READY pgs=3339 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0097d0 tx=0x7fe9ac02b810 comp rx=0 tx=0).stop 2024-02-06T22:17:58.347 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.343+0000 7fe9c6de6640 1 -- 172.21.15.120:0/1982384033 shutdown_connections 2024-02-06T22:17:58.347 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.343+0000 7fe9c6de6640 1 --2- 172.21.15.120:0/1982384033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9c0140050 0x7fe9c00a9860 unknown :-1 s=CLOSED pgs=3339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:58.347 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.343+0000 7fe9c6de6640 1 -- 172.21.15.120:0/1982384033 >> 172.21.15.120:0/1982384033 conn(0x7fe9c00b8660 msgr2=0x7fe9c00b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:17:58.347 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.343+0000 7fe9c6de6640 1 -- 172.21.15.120:0/1982384033 shutdown_connections 2024-02-06T22:17:58.347 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.343+0000 7fe9c6de6640 1 -- 172.21.15.120:0/1982384033 wait complete. 2024-02-06T22:17:58.350 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.343+0000 7fe9c6de6640 10 monclient: build_initial_monmap 2024-02-06T22:17:58.351 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.343+0000 7fe9c6de6640 10 monclient: monmap: 2024-02-06T22:17:58.351 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:17:58.351 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:17:58.351 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:17:58.351 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:17:58.351 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:17:58.351 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:58.351 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:17:58.351 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:17:58.351 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:17:58.351 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:58.351 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.344+0000 7fe9c6de6640 1 librados: starting msgr at 2024-02-06T22:17:58.351 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.344+0000 7fe9c6de6640 1 librados: starting objecter 2024-02-06T22:17:58.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.344+0000 7fe9c6de6640 1 Processor -- start 2024-02-06T22:17:58.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.344+0000 7fe9c6de6640 1 -- start start 2024-02-06T22:17:58.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.344+0000 7fe9c6de6640 1 librados: setting wanted keys 2024-02-06T22:17:58.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.344+0000 7fe9c6de6640 1 librados: calling monclient init 2024-02-06T22:17:58.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.344+0000 7fe9c6de6640 10 monclient: init 2024-02-06T22:17:58.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 10 monclient: _reopen_session rank -1 2024-02-06T22:17:58.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T22:17:58.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9c00b54b0 0x7fe9c0146620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:58.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c00b54b0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:17:58.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c0140050 0x7fe9c0132390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:58.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c0140050 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:17:58.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9c0142680 0x7fe9c01439c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:58.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c0142680 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c01328d0 con 0x7fe9c0140050 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0133660 con 0x7fe9c00b54b0 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0133940 con 0x7fe9c0142680 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 10 monclient(hunting): _renew_subs 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c6de6640 10 monclient(hunting): authenticate will time out at 3508.682622s 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9bffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c0140050 0x7fe9c0132390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c5383640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9c0142680 0x7fe9c01439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c4b82640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9c00b54b0 0x7fe9c0146620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9bffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c0140050 0x7fe9c0132390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:41512/0 (socket says 172.21.15.120:41512) 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9bffff640 1 -- 172.21.15.120:0/2986677044 learned_addr learned my addr 172.21.15.120:0/2986677044 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9bffff640 10 monclient(hunting): get_auth_request con 0x7fe9c0140050 auth_method 0 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:58.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9bffff640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:58.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9bffff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:58.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c5383640 10 monclient(hunting): get_auth_request con 0x7fe9c0142680 auth_method 0 2024-02-06T22:17:58.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c5383640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:58.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c5383640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:58.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.345+0000 7fe9c5383640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:58.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.346+0000 7fe9c4b82640 10 monclient(hunting): get_auth_request con 0x7fe9c00b54b0 auth_method 0 2024-02-06T22:17:58.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.346+0000 7fe9c4b82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:58.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.346+0000 7fe9c4b82640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:58.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.346+0000 7fe9c4b82640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:58.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.346+0000 7fe9bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:58.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.346+0000 7fe9bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:58.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.346+0000 7fe9bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:58.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.346+0000 7fe9c5383640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:58.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.346+0000 7fe9c5383640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:58.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.346+0000 7fe9c5383640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:58.355 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.346+0000 7fe9c4b82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:58.355 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.346+0000 7fe9c4b82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:58.355 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.346+0000 7fe9c4b82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:58.355 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.349+0000 7fe9c4b82640 10 monclient(hunting): handle_auth_done global_id 21019 payload 824 2024-02-06T22:17:58.355 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.349+0000 7fe9c4b82640 1 -- 172.21.15.120:0/2986677044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9c0142680 msgr2=0x7fe9c01439c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:58.355 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.349+0000 7fe9c4b82640 1 --2- 172.21.15.120:0/2986677044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9c0142680 0x7fe9c01439c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:58.355 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.349+0000 7fe9c4b82640 1 -- 172.21.15.120:0/2986677044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c0140050 msgr2=0x7fe9c0132390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:58.355 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.349+0000 7fe9c4b82640 1 --2- 172.21.15.120:0/2986677044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c0140050 0x7fe9c0132390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:58.355 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.349+0000 7fe9c4b82640 10 monclient: _finish_hunting 0 2024-02-06T22:17:58.355 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.349+0000 7fe9c4b82640 1 monclient: found mon.noname-b 2024-02-06T22:17:58.355 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.349+0000 7fe9c4b82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:17:58.355 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.349+0000 7fe9c4b82640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T22:17:58.355 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.349+0000 7fe9c4b82640 1 -- 172.21.15.120:0/2986677044 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac009400 con 0x7fe9c00b54b0 2024-02-06T22:17:58.355 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.349+0000 7fe9bffff640 1 --2- 172.21.15.120:0/2986677044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c0140050 0x7fe9c0132390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-06T22:17:58.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.351+0000 7fe9c4b82640 1 --2- 172.21.15.120:0/2986677044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9c00b54b0 0x7fe9c0146620 secure :-1 s=READY pgs=5086 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400c3e0 tx=0x7fe9b400c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:17:58.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.351+0000 7fe9bd7fa640 10 client.?.objecter ms_handle_connect 0x7fe9c00b54b0 2024-02-06T22:17:58.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.351+0000 7fe9bd7fa640 10 client.?.objecter resend_mon_ops 2024-02-06T22:17:58.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9bd7fa640 1 -- 172.21.15.120:0/2986677044 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4005ce0 con 0x7fe9c00b54b0 2024-02-06T22:17:58.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:58.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9bd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:17:58.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9bd7fa640 10 monclient: dump: 2024-02-06T22:17:58.357 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:58.357 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:58.357 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:58.358 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:58.358 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:58.358 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:58.358 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:58.358 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:58.358 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:58.358 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:58.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9bd7fa640 10 monclient: _finish_auth 0 2024-02-06T22:17:58.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9bd7fa640 10 monclient: _check_auth_tickets 2024-02-06T22:17:58.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:17:58.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9bd7fa640 1 -- 172.21.15.120:0/2986677044 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b4005e60 con 0x7fe9c00b54b0 2024-02-06T22:17:58.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:17:58.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9bd7fa640 1 -- 172.21.15.120:0/2986677044 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4007310 con 0x7fe9c00b54b0 2024-02-06T22:17:58.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9c6de6640 5 monclient: authenticate success, global_id 21019 2024-02-06T22:17:58.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9c6de6640 10 monclient: _renew_subs 2024-02-06T22:17:58.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9c6de6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:17:58.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9c6de6640 1 -- 172.21.15.120:0/2986677044 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c0145b40 con 0x7fe9c00b54b0 2024-02-06T22:17:58.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9beffd640 4 set_mon_vals no callback set 2024-02-06T22:17:58.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:58.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9bd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T22:17:58.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9bd7fa640 10 monclient: dump: 2024-02-06T22:17:58.359 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:58.359 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:58.359 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:58.359 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:58.359 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:58.359 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:58.360 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:58.360 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:58.360 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:58.360 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:58.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9c6de6640 10 client.21019.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:17:58.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9c6de6640 10 monclient: _renew_subs 2024-02-06T22:17:58.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9c6de6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:17:58.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9c6de6640 1 -- 172.21.15.120:0/2986677044 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c0143f00 con 0x7fe9c00b54b0 2024-02-06T22:17:58.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.355+0000 7fe9c6de6640 1 librados: init done 2024-02-06T22:17:58.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.356+0000 7fe9c6de6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T22:17:58.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.356+0000 7fe9c6de6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:17:58.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.356+0000 7fe9c6de6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:17:58.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.356+0000 7fe9c6de6640 1 -- 172.21.15.120:0/2986677044 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c004ef90 con 0x7fe9c00b54b0 2024-02-06T22:17:58.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.607+0000 7fe9bd7fa640 1 -- 172.21.15.120:0/2986677044 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe9b4007470 con 0x7fe9c00b54b0 2024-02-06T22:17:58.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.607+0000 7fe9bd7fa640 1 --2- 172.21.15.120:0/2986677044 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe9a406d1a0 0x7fe9a406f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:58.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.607+0000 7fe9bffff640 1 --2- 172.21.15.120:0/2986677044 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe9a406d1a0 0x7fe9a406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:58.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.607+0000 7fe9bffff640 10 monclient: get_auth_request con 0x7fe9a406d1a0 auth_method 0 2024-02-06T22:17:58.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.608+0000 7fe9bd7fa640 10 client.21019.objecter ms_handle_connect 0x7fe9a406d1a0 2024-02-06T22:17:58.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.608+0000 7fe9bffff640 1 --2- 172.21.15.120:0/2986677044 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe9a406d1a0 0x7fe9a406f660 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac000c00 tx=0x7fe9ac03d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:17:58.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.612+0000 7fe9bd7fa640 1 -- 172.21.15.120:0/2986677044 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(1345..1345 src has 1047..1345) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fe9b40583c0 con 0x7fe9c00b54b0 2024-02-06T22:17:58.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.612+0000 7fe9bd7fa640 10 client.21019.objecter ms_dispatch 0x7fe9c0000b90 osd_map(1345..1345 src has 1047..1345) v4 2024-02-06T22:17:58.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.612+0000 7fe9bd7fa640 3 client.21019.objecter handle_osd_map got epochs [1345,1345] > 0 2024-02-06T22:17:58.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.612+0000 7fe9bd7fa640 3 client.21019.objecter handle_osd_map decoding full epoch 1345 2024-02-06T22:17:58.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.612+0000 7fe9bd7fa640 20 client.21019.objecter dump_active .. 0 homeless 2024-02-06T22:17:58.623 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.622+0000 7fe9bd7fa640 1 -- 172.21.15.120:0/2986677044 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe9b4054660 con 0x7fe9c00b54b0 2024-02-06T22:17:58.623 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.622+0000 7fe9bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:17:58.623 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.622+0000 7fe9bd7fa640 10 monclient: _finish_command 1 = system:0 2024-02-06T22:17:58.873 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.871+0000 7fe9c6de6640 1 -- 172.21.15.120:0/2986677044 --> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe9c013cb80 con 0x7fe9a406d1a0 2024-02-06T22:17:58.877 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.876+0000 7fe9bd7fa640 1 -- 172.21.15.120:0/2986677044 <== mgr.4100 v2:172.21.15.135:6800/315664144 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151675 (secure 0 0 0) 0x7fe9c013cb80 con 0x7fe9a406d1a0 2024-02-06T22:17:58.879 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T22:17:58.879 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2024-02-06T22:17:58.881 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 10 librados: watch_flush enter 2024-02-06T22:17:58.881 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 10 librados: watch_flush exit 2024-02-06T22:17:58.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 20 client.21019.objecter shutdown clearing up homeless session... 2024-02-06T22:17:58.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 10 client.21019.objecter successfully canceled tick 2024-02-06T22:17:58.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 1 -- 172.21.15.120:0/2986677044 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe9a406d1a0 msgr2=0x7fe9a406f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:58.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 1 --2- 172.21.15.120:0/2986677044 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe9a406d1a0 0x7fe9a406f660 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac000c00 tx=0x7fe9ac03d040 comp rx=0 tx=0).stop 2024-02-06T22:17:58.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 10 monclient: shutdown 2024-02-06T22:17:58.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:17:58.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 1 -- 172.21.15.120:0/2986677044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9c00b54b0 msgr2=0x7fe9c0146620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:58.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 1 --2- 172.21.15.120:0/2986677044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9c00b54b0 0x7fe9c0146620 secure :-1 s=READY pgs=5086 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400c3e0 tx=0x7fe9b400c8a0 comp rx=0 tx=0).stop 2024-02-06T22:17:58.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 1 -- 172.21.15.120:0/2986677044 shutdown_connections 2024-02-06T22:17:58.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 1 --2- 172.21.15.120:0/2986677044 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe9a406d1a0 0x7fe9a406f660 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:58.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 1 --2- 172.21.15.120:0/2986677044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9c0142680 0x7fe9c01439c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:58.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 1 --2- 172.21.15.120:0/2986677044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c0140050 0x7fe9c0132390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:58.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 1 --2- 172.21.15.120:0/2986677044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9c00b54b0 0x7fe9c0146620 unknown :-1 s=CLOSED pgs=5086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:58.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 1 -- 172.21.15.120:0/2986677044 >> 172.21.15.120:0/2986677044 conn(0x7fe9c00b8660 msgr2=0x7fe9c00744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:17:58.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 1 -- 172.21.15.120:0/2986677044 shutdown_connections 2024-02-06T22:17:58.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 1 -- 172.21.15.120:0/2986677044 wait complete. 2024-02-06T22:17:58.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:58.879+0000 7fe9c6de6640 1 librados: shutdown 2024-02-06T22:17:58.894 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":2448,"stamp":"2024-02-06T22:17:58.290638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":583016,"kb_used_data":40648,"kb_used_omap":12,"kb_used_meta":542259,"kb_avail":754391704,"statfs":{"total":773094113280,"available":772497104896,"internally_reserved":0,"allocated":41623552,"data_stored":19797914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":555273776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":156,"apply_latency_ms":156,"commit_latency_ns":156000000,"apply_latency_ns":156000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.938777"},"pg_stats":[{"pgid":"12.1c","version":"0'0","reported_seq":2590,"reported_epoch":1344,"state":"active+clean","last_fresh":"2024-02-06T22:17:55.636349+0000","last_change":"2024-02-06T21:42:50.083729+0000","last_active":"2024-02-06T22:17:55.636349+0000","last_peered":"2024-02-06T22:17:55.636349+0000","last_clean":"2024-02-06T22:17:55.636349+0000","last_became_active":"2024-02-06T21:42:50.078423+0000","last_became_peered":"2024-02-06T21:42:50.078423+0000","last_unstale":"2024-02-06T22:17:55.636349+0000","last_undegraded":"2024-02-06T22:17:55.636349+0000","last_fullsized":"2024-02-06T22:17:55.636349+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:44:16.757427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":2580,"reported_epoch":1343,"state":"active+clean","last_fresh":"2024-02-06T22:17:54.405293+0000","last_change":"2024-02-06T21:43:19.779411+0000","last_active":"2024-02-06T22:17:54.405293+0000","last_peered":"2024-02-06T22:17:54.405293+0000","last_clean":"2024-02-06T22:17:54.405293+0000","last_became_active":"2024-02-06T21:43:19.773686+0000","last_became_peered":"2024-02-06T21:43:19.773686+0000","last_unstale":"2024-02-06T22:17:54.405293+0000","last_undegraded":"2024-02-06T22:17:54.405293+0000","last_fullsized":"2024-02-06T22:17:54.405293+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:54:24.026659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":2585,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.382909+0000","last_change":"2024-02-06T21:42:53.308445+0000","last_active":"2024-02-06T22:17:53.382909+0000","last_peered":"2024-02-06T22:17:53.382909+0000","last_clean":"2024-02-06T22:17:53.382909+0000","last_became_active":"2024-02-06T21:42:53.291194+0000","last_became_peered":"2024-02-06T21:42:53.291194+0000","last_unstale":"2024-02-06T22:17:53.382909+0000","last_undegraded":"2024-02-06T22:17:53.382909+0000","last_fullsized":"2024-02-06T22:17:53.382909+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T00:08:35.742342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":2575,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.511147+0000","last_change":"2024-02-06T21:42:45.080259+0000","last_active":"2024-02-06T22:17:53.511147+0000","last_peered":"2024-02-06T22:17:53.511147+0000","last_clean":"2024-02-06T22:17:53.511147+0000","last_became_active":"2024-02-06T21:42:45.065933+0000","last_became_peered":"2024-02-06T21:42:45.065933+0000","last_unstale":"2024-02-06T22:17:53.511147+0000","last_undegraded":"2024-02-06T22:17:53.511147+0000","last_fullsized":"2024-02-06T22:17:53.511147+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:44:14.566236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":2575,"reported_epoch":1344,"state":"active+clean","last_fresh":"2024-02-06T22:17:56.140689+0000","last_change":"2024-02-06T21:42:58.554516+0000","last_active":"2024-02-06T22:17:56.140689+0000","last_peered":"2024-02-06T22:17:56.140689+0000","last_clean":"2024-02-06T22:17:56.140689+0000","last_became_active":"2024-02-06T21:42:58.548361+0000","last_became_peered":"2024-02-06T21:42:58.548361+0000","last_unstale":"2024-02-06T22:17:56.140689+0000","last_undegraded":"2024-02-06T22:17:56.140689+0000","last_fullsized":"2024-02-06T22:17:56.140689+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T07:18:58.848738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"124'7","reported_seq":2596,"reported_epoch":1345,"state":"active+clean","last_fresh":"2024-02-06T22:17:56.420799+0000","last_change":"2024-02-06T21:43:15.527724+0000","last_active":"2024-02-06T22:17:56.420799+0000","last_peered":"2024-02-06T22:17:56.420799+0000","last_clean":"2024-02-06T22:17:56.420799+0000","last_became_active":"2024-02-06T21:43:15.497081+0000","last_became_peered":"2024-02-06T21:43:15.497081+0000","last_unstale":"2024-02-06T22:17:56.420799+0000","last_undegraded":"2024-02-06T22:17:56.420799+0000","last_fullsized":"2024-02-06T22:17:56.420799+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:53:37.062280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":2586,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.365646+0000","last_change":"2024-02-06T21:43:13.230133+0000","last_active":"2024-02-06T22:17:53.365646+0000","last_peered":"2024-02-06T22:17:53.365646+0000","last_clean":"2024-02-06T22:17:53.365646+0000","last_became_active":"2024-02-06T21:43:13.108638+0000","last_became_peered":"2024-02-06T21:43:13.108638+0000","last_unstale":"2024-02-06T22:17:53.365646+0000","last_undegraded":"2024-02-06T22:17:53.365646+0000","last_fullsized":"2024-02-06T22:17:53.365646+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:24:54.727933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":2593,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.377205+0000","last_change":"2024-02-06T21:43:06.276185+0000","last_active":"2024-02-06T22:17:53.377205+0000","last_peered":"2024-02-06T22:17:53.377205+0000","last_clean":"2024-02-06T22:17:53.377205+0000","last_became_active":"2024-02-06T21:43:06.270947+0000","last_became_peered":"2024-02-06T21:43:06.270947+0000","last_unstale":"2024-02-06T22:17:53.377205+0000","last_undegraded":"2024-02-06T22:17:53.377205+0000","last_fullsized":"2024-02-06T22:17:53.377205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:27:26.086535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":2586,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.379590+0000","last_change":"2024-02-06T21:42:53.563677+0000","last_active":"2024-02-06T22:17:53.379590+0000","last_peered":"2024-02-06T22:17:53.379590+0000","last_clean":"2024-02-06T22:17:53.379590+0000","last_became_active":"2024-02-06T21:42:53.535059+0000","last_became_peered":"2024-02-06T21:42:53.535059+0000","last_unstale":"2024-02-06T22:17:53.379590+0000","last_undegraded":"2024-02-06T22:17:53.379590+0000","last_fullsized":"2024-02-06T22:17:53.379590+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T06:13:12.610796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":2577,"reported_epoch":1344,"state":"active+clean","last_fresh":"2024-02-06T22:17:56.152224+0000","last_change":"2024-02-06T21:42:46.177846+0000","last_active":"2024-02-06T22:17:56.152224+0000","last_peered":"2024-02-06T22:17:56.152224+0000","last_clean":"2024-02-06T22:17:56.152224+0000","last_became_active":"2024-02-06T21:42:46.168640+0000","last_became_peered":"2024-02-06T21:42:46.168640+0000","last_unstale":"2024-02-06T22:17:56.152224+0000","last_undegraded":"2024-02-06T22:17:56.152224+0000","last_fullsized":"2024-02-06T22:17:56.152224+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:37:19.536133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":2572,"reported_epoch":1343,"state":"active+clean","last_fresh":"2024-02-06T22:17:54.417919+0000","last_change":"2024-02-06T21:43:27.891680+0000","last_active":"2024-02-06T22:17:54.417919+0000","last_peered":"2024-02-06T22:17:54.417919+0000","last_clean":"2024-02-06T22:17:54.417919+0000","last_became_active":"2024-02-06T21:43:27.883526+0000","last_became_peered":"2024-02-06T21:43:27.883526+0000","last_unstale":"2024-02-06T22:17:54.417919+0000","last_undegraded":"2024-02-06T22:17:54.417919+0000","last_fullsized":"2024-02-06T22:17:54.417919+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:21:41.402059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":2581,"reported_epoch":1345,"state":"active+clean","last_fresh":"2024-02-06T22:17:57.449814+0000","last_change":"2024-02-06T21:42:45.882433+0000","last_active":"2024-02-06T22:17:57.449814+0000","last_peered":"2024-02-06T22:17:57.449814+0000","last_clean":"2024-02-06T22:17:57.449814+0000","last_became_active":"2024-02-06T21:42:45.868987+0000","last_became_peered":"2024-02-06T21:42:45.868987+0000","last_unstale":"2024-02-06T22:17:57.449814+0000","last_undegraded":"2024-02-06T22:17:57.449814+0000","last_fullsized":"2024-02-06T22:17:57.449814+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T00:26:41.132150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":2572,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.498068+0000","last_change":"2024-02-06T21:42:59.955119+0000","last_active":"2024-02-06T22:17:53.498068+0000","last_peered":"2024-02-06T22:17:53.498068+0000","last_clean":"2024-02-06T22:17:53.498068+0000","last_became_active":"2024-02-06T21:42:59.949344+0000","last_became_peered":"2024-02-06T21:42:59.949344+0000","last_unstale":"2024-02-06T22:17:53.498068+0000","last_undegraded":"2024-02-06T22:17:53.498068+0000","last_fullsized":"2024-02-06T22:17:53.498068+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T06:07:53.514679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'76","reported_seq":2702,"reported_epoch":1345,"state":"active+clean","last_fresh":"2024-02-06T22:17:56.410085+0000","last_change":"2024-02-06T21:41:31.339576+0000","last_active":"2024-02-06T22:17:56.410085+0000","last_peered":"2024-02-06T22:17:56.410085+0000","last_clean":"2024-02-06T22:17:56.410085+0000","last_became_active":"2024-02-06T21:41:31.305766+0000","last_became_peered":"2024-02-06T21:41:31.305766+0000","last_unstale":"2024-02-06T22:17:56.410085+0000","last_undegraded":"2024-02-06T22:17:56.410085+0000","last_fullsized":"2024-02-06T22:17:56.410085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:20.409569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:20.409569+0000","last_clean_scrub_stamp":"2024-02-06T21:41:20.409569+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:24:57.767580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2586,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.504352+0000","last_change":"2024-02-06T21:41:36.168152+0000","last_active":"2024-02-06T22:17:53.504352+0000","last_peered":"2024-02-06T22:17:53.504352+0000","last_clean":"2024-02-06T22:17:53.504352+0000","last_became_active":"2024-02-06T21:41:36.162988+0000","last_became_peered":"2024-02-06T21:41:36.162988+0000","last_unstale":"2024-02-06T22:17:53.504352+0000","last_undegraded":"2024-02-06T22:17:53.504352+0000","last_fullsized":"2024-02-06T22:17:53.504352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T00:47:41.526670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":2580,"reported_epoch":1345,"state":"active+clean","last_fresh":"2024-02-06T22:17:57.440201+0000","last_change":"2024-02-06T21:42:56.031740+0000","last_active":"2024-02-06T22:17:57.440201+0000","last_peered":"2024-02-06T22:17:57.440201+0000","last_clean":"2024-02-06T22:17:57.440201+0000","last_became_active":"2024-02-06T21:42:56.023045+0000","last_became_peered":"2024-02-06T21:42:56.023045+0000","last_unstale":"2024-02-06T22:17:57.440201+0000","last_undegraded":"2024-02-06T22:17:57.440201+0000","last_fullsized":"2024-02-06T22:17:57.440201+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:19:09.477603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2587,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.467482+0000","last_change":"2024-02-06T21:41:35.688242+0000","last_active":"2024-02-06T22:17:53.467482+0000","last_peered":"2024-02-06T22:17:53.467482+0000","last_clean":"2024-02-06T22:17:53.467482+0000","last_became_active":"2024-02-06T21:41:35.642164+0000","last_became_peered":"2024-02-06T21:41:35.642164+0000","last_unstale":"2024-02-06T22:17:53.467482+0000","last_undegraded":"2024-02-06T22:17:53.467482+0000","last_fullsized":"2024-02-06T22:17:53.467482+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:50:03.838525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":2569,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.512725+0000","last_change":"2024-02-06T21:43:05.426147+0000","last_active":"2024-02-06T22:17:53.512725+0000","last_peered":"2024-02-06T22:17:53.512725+0000","last_clean":"2024-02-06T22:17:53.512725+0000","last_became_active":"2024-02-06T21:43:05.414230+0000","last_became_peered":"2024-02-06T21:43:05.414230+0000","last_unstale":"2024-02-06T22:17:53.512725+0000","last_undegraded":"2024-02-06T22:17:53.512725+0000","last_fullsized":"2024-02-06T22:17:53.512725+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:04:00.128718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2592,"reported_epoch":1345,"state":"active+clean","last_fresh":"2024-02-06T22:17:57.445851+0000","last_change":"2024-02-06T21:41:35.636555+0000","last_active":"2024-02-06T22:17:57.445851+0000","last_peered":"2024-02-06T22:17:57.445851+0000","last_clean":"2024-02-06T22:17:57.445851+0000","last_became_active":"2024-02-06T21:41:35.603135+0000","last_became_peered":"2024-02-06T21:41:35.603135+0000","last_unstale":"2024-02-06T22:17:57.445851+0000","last_undegraded":"2024-02-06T22:17:57.445851+0000","last_fullsized":"2024-02-06T22:17:57.445851+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T03:13:14.699865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":2581,"reported_epoch":1345,"state":"active+clean","last_fresh":"2024-02-06T22:17:57.436903+0000","last_change":"2024-02-06T21:42:49.418971+0000","last_active":"2024-02-06T22:17:57.436903+0000","last_peered":"2024-02-06T22:17:57.436903+0000","last_clean":"2024-02-06T22:17:57.436903+0000","last_became_active":"2024-02-06T21:42:49.409862+0000","last_became_peered":"2024-02-06T21:42:49.409862+0000","last_unstale":"2024-02-06T22:17:57.436903+0000","last_undegraded":"2024-02-06T22:17:57.436903+0000","last_fullsized":"2024-02-06T22:17:57.436903+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:08:26.530828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2587,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.487524+0000","last_change":"2024-02-06T21:41:35.892547+0000","last_active":"2024-02-06T22:17:53.487524+0000","last_peered":"2024-02-06T22:17:53.487524+0000","last_clean":"2024-02-06T22:17:53.487524+0000","last_became_active":"2024-02-06T21:41:35.877657+0000","last_became_peered":"2024-02-06T21:41:35.877657+0000","last_unstale":"2024-02-06T22:17:53.487524+0000","last_undegraded":"2024-02-06T22:17:53.487524+0000","last_fullsized":"2024-02-06T22:17:53.487524+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:23:28.851375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":2593,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.383234+0000","last_change":"2024-02-06T21:42:56.499357+0000","last_active":"2024-02-06T22:17:53.383234+0000","last_peered":"2024-02-06T22:17:53.383234+0000","last_clean":"2024-02-06T22:17:53.383234+0000","last_became_active":"2024-02-06T21:42:56.473273+0000","last_became_peered":"2024-02-06T21:42:56.473273+0000","last_unstale":"2024-02-06T22:17:53.383234+0000","last_undegraded":"2024-02-06T22:17:53.383234+0000","last_fullsized":"2024-02-06T22:17:53.383234+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:45:45.452902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2602,"reported_epoch":1344,"state":"active+clean","last_fresh":"2024-02-06T22:17:55.639167+0000","last_change":"2024-02-06T21:41:35.918723+0000","last_active":"2024-02-06T22:17:55.639167+0000","last_peered":"2024-02-06T22:17:55.639167+0000","last_clean":"2024-02-06T22:17:55.639167+0000","last_became_active":"2024-02-06T21:41:35.891748+0000","last_became_peered":"2024-02-06T21:41:35.891748+0000","last_unstale":"2024-02-06T22:17:55.639167+0000","last_undegraded":"2024-02-06T22:17:55.639167+0000","last_fullsized":"2024-02-06T22:17:55.639167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T03:59:11.081636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":2581,"reported_epoch":1345,"state":"active+clean","last_fresh":"2024-02-06T22:17:57.433904+0000","last_change":"2024-02-06T21:42:49.677946+0000","last_active":"2024-02-06T22:17:57.433904+0000","last_peered":"2024-02-06T22:17:57.433904+0000","last_clean":"2024-02-06T22:17:57.433904+0000","last_became_active":"2024-02-06T21:42:49.671861+0000","last_became_peered":"2024-02-06T21:42:49.671861+0000","last_unstale":"2024-02-06T22:17:57.433904+0000","last_undegraded":"2024-02-06T22:17:57.433904+0000","last_fullsized":"2024-02-06T22:17:57.433904+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:43:48.362335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2586,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.499796+0000","last_change":"2024-02-06T21:41:35.766413+0000","last_active":"2024-02-06T22:17:53.499796+0000","last_peered":"2024-02-06T22:17:53.499796+0000","last_clean":"2024-02-06T22:17:53.499796+0000","last_became_active":"2024-02-06T21:41:35.761324+0000","last_became_peered":"2024-02-06T21:41:35.761324+0000","last_unstale":"2024-02-06T22:17:53.499796+0000","last_undegraded":"2024-02-06T22:17:53.499796+0000","last_fullsized":"2024-02-06T22:17:53.499796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:28:07.018583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":2582,"reported_epoch":1345,"state":"active+clean","last_fresh":"2024-02-06T22:17:57.424542+0000","last_change":"2024-02-06T21:42:44.700963+0000","last_active":"2024-02-06T22:17:57.424542+0000","last_peered":"2024-02-06T22:17:57.424542+0000","last_clean":"2024-02-06T22:17:57.424542+0000","last_became_active":"2024-02-06T21:42:44.691217+0000","last_became_peered":"2024-02-06T21:42:44.691217+0000","last_unstale":"2024-02-06T22:17:57.424542+0000","last_undegraded":"2024-02-06T22:17:57.424542+0000","last_fullsized":"2024-02-06T22:17:57.424542+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:15:55.323353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2609,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.390113+0000","last_change":"2024-02-06T21:41:36.193098+0000","last_active":"2024-02-06T22:17:53.390113+0000","last_peered":"2024-02-06T22:17:53.390113+0000","last_clean":"2024-02-06T22:17:53.390113+0000","last_became_active":"2024-02-06T21:41:36.160648+0000","last_became_peered":"2024-02-06T21:41:36.160648+0000","last_unstale":"2024-02-06T22:17:53.390113+0000","last_undegraded":"2024-02-06T22:17:53.390113+0000","last_fullsized":"2024-02-06T22:17:53.390113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T09:01:03.161738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":2582,"reported_epoch":1343,"state":"active+clean","last_fresh":"2024-02-06T22:17:54.409582+0000","last_change":"2024-02-06T21:42:52.618077+0000","last_active":"2024-02-06T22:17:54.409582+0000","last_peered":"2024-02-06T22:17:54.409582+0000","last_clean":"2024-02-06T22:17:54.409582+0000","last_became_active":"2024-02-06T21:42:52.611794+0000","last_became_peered":"2024-02-06T21:42:52.611794+0000","last_unstale":"2024-02-06T22:17:54.409582+0000","last_undegraded":"2024-02-06T22:17:54.409582+0000","last_fullsized":"2024-02-06T22:17:54.409582+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:25:35.938955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2583,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.514531+0000","last_change":"2024-02-06T21:41:36.144443+0000","last_active":"2024-02-06T22:17:53.514531+0000","last_peered":"2024-02-06T22:17:53.514531+0000","last_clean":"2024-02-06T22:17:53.514531+0000","last_became_active":"2024-02-06T21:41:36.139312+0000","last_became_peered":"2024-02-06T21:41:36.139312+0000","last_unstale":"2024-02-06T22:17:53.514531+0000","last_undegraded":"2024-02-06T22:17:53.514531+0000","last_fullsized":"2024-02-06T22:17:53.514531+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:47:25.348858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":2576,"reported_epoch":1344,"state":"active+clean","last_fresh":"2024-02-06T22:17:56.145480+0000","last_change":"2024-02-06T21:42:49.714094+0000","last_active":"2024-02-06T22:17:56.145480+0000","last_peered":"2024-02-06T22:17:56.145480+0000","last_clean":"2024-02-06T22:17:56.145480+0000","last_became_active":"2024-02-06T21:42:49.708820+0000","last_became_peered":"2024-02-06T21:42:49.708820+0000","last_unstale":"2024-02-06T22:17:56.145480+0000","last_undegraded":"2024-02-06T22:17:56.145480+0000","last_fullsized":"2024-02-06T22:17:56.145480+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:55:54.917799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":2584,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.362656+0000","last_change":"2024-02-06T21:43:13.244098+0000","last_active":"2024-02-06T22:17:53.362656+0000","last_peered":"2024-02-06T22:17:53.362656+0000","last_clean":"2024-02-06T22:17:53.362656+0000","last_became_active":"2024-02-06T21:43:13.123786+0000","last_became_peered":"2024-02-06T21:43:13.123786+0000","last_unstale":"2024-02-06T22:17:53.362656+0000","last_undegraded":"2024-02-06T22:17:53.362656+0000","last_fullsized":"2024-02-06T22:17:53.362656+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T02:53:35.916469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":2576,"reported_epoch":1345,"state":"active+clean","last_fresh":"2024-02-06T22:17:57.442804+0000","last_change":"2024-02-06T21:42:56.872139+0000","last_active":"2024-02-06T22:17:57.442804+0000","last_peered":"2024-02-06T22:17:57.442804+0000","last_clean":"2024-02-06T22:17:57.442804+0000","last_became_active":"2024-02-06T21:42:56.865960+0000","last_became_peered":"2024-02-06T21:42:56.865960+0000","last_unstale":"2024-02-06T22:17:57.442804+0000","last_undegraded":"2024-02-06T22:17:57.442804+0000","last_fullsized":"2024-02-06T22:17:57.442804+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:13:16.665309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":2580,"reported_epoch":1345,"state":"active+clean","last_fresh":"2024-02-06T22:17:57.426591+0000","last_change":"2024-02-06T21:42:45.712485+0000","last_active":"2024-02-06T22:17:57.426591+0000","last_peered":"2024-02-06T22:17:57.426591+0000","last_clean":"2024-02-06T22:17:57.426591+0000","last_became_active":"2024-02-06T21:42:45.706432+0000","last_became_peered":"2024-02-06T21:42:45.706432+0000","last_unstale":"2024-02-06T22:17:57.426591+0000","last_undegraded":"2024-02-06T22:17:57.426591+0000","last_fullsized":"2024-02-06T22:17:57.426591+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:46:59.377729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":2577,"reported_epoch":1345,"state":"active+clean","last_fresh":"2024-02-06T22:17:56.426278+0000","last_change":"2024-02-06T21:43:08.879863+0000","last_active":"2024-02-06T22:17:56.426278+0000","last_peered":"2024-02-06T22:17:56.426278+0000","last_clean":"2024-02-06T22:17:56.426278+0000","last_became_active":"2024-02-06T21:43:08.871254+0000","last_became_peered":"2024-02-06T21:43:08.871254+0000","last_unstale":"2024-02-06T22:17:56.426278+0000","last_undegraded":"2024-02-06T22:17:56.426278+0000","last_fullsized":"2024-02-06T22:17:56.426278+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:02:54.631582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":2582,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.384957+0000","last_change":"2024-02-06T21:42:55.060952+0000","last_active":"2024-02-06T22:17:53.384957+0000","last_peered":"2024-02-06T22:17:53.384957+0000","last_clean":"2024-02-06T22:17:53.384957+0000","last_became_active":"2024-02-06T21:42:55.034062+0000","last_became_peered":"2024-02-06T21:42:55.034062+0000","last_unstale":"2024-02-06T22:17:53.384957+0000","last_undegraded":"2024-02-06T22:17:53.384957+0000","last_fullsized":"2024-02-06T22:17:53.384957+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T07:52:53.348265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":2580,"reported_epoch":1345,"state":"active+clean","last_fresh":"2024-02-06T22:17:56.417911+0000","last_change":"2024-02-06T21:43:15.867032+0000","last_active":"2024-02-06T22:17:56.417911+0000","last_peered":"2024-02-06T22:17:56.417911+0000","last_clean":"2024-02-06T22:17:56.417911+0000","last_became_active":"2024-02-06T21:43:15.861140+0000","last_became_peered":"2024-02-06T21:43:15.861140+0000","last_unstale":"2024-02-06T22:17:56.417911+0000","last_undegraded":"2024-02-06T22:17:56.417911+0000","last_fullsized":"2024-02-06T22:17:56.417911+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:00:40.646612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":2577,"reported_epoch":1344,"state":"active+clean","last_fresh":"2024-02-06T22:17:56.150028+0000","last_change":"2024-02-06T21:42:55.175403+0000","last_active":"2024-02-06T22:17:56.150028+0000","last_peered":"2024-02-06T22:17:56.150028+0000","last_clean":"2024-02-06T22:17:56.150028+0000","last_became_active":"2024-02-06T21:42:55.168677+0000","last_became_peered":"2024-02-06T21:42:55.168677+0000","last_unstale":"2024-02-06T22:17:56.150028+0000","last_undegraded":"2024-02-06T22:17:56.150028+0000","last_fullsized":"2024-02-06T22:17:56.150028+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T07:31:47.782588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":2592,"reported_epoch":1344,"state":"active+clean","last_fresh":"2024-02-06T22:17:55.625773+0000","last_change":"2024-02-06T21:42:52.000053+0000","last_active":"2024-02-06T22:17:55.625773+0000","last_peered":"2024-02-06T22:17:55.625773+0000","last_clean":"2024-02-06T22:17:55.625773+0000","last_became_active":"2024-02-06T21:42:51.993812+0000","last_became_peered":"2024-02-06T21:42:51.993812+0000","last_unstale":"2024-02-06T22:17:55.625773+0000","last_undegraded":"2024-02-06T22:17:55.625773+0000","last_fullsized":"2024-02-06T22:17:55.625773+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:44:21.306346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":2572,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.382261+0000","last_change":"2024-02-06T21:44:01.005089+0000","last_active":"2024-02-06T22:17:53.382261+0000","last_peered":"2024-02-06T22:17:53.382261+0000","last_clean":"2024-02-06T22:17:53.382261+0000","last_became_active":"2024-02-06T21:44:00.992836+0000","last_became_peered":"2024-02-06T21:44:00.992836+0000","last_unstale":"2024-02-06T22:17:53.382261+0000","last_undegraded":"2024-02-06T22:17:53.382261+0000","last_fullsized":"2024-02-06T22:17:53.382261+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:34:43.433551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":2580,"reported_epoch":1342,"state":"active+clean","last_fresh":"2024-02-06T22:17:53.359066+0000","last_change":"2024-02-06T21:43:14.066712+0000","last_active":"2024-02-06T22:17:53.359066+0000","last_peered":"2024-02-06T22:17:53.359066+0000","last_clean":"2024-02-06T22:17:53.359066+0000","last_became_active":"2024-02-06T21:43:14.057258+0000","last_became_peered":"2024-02-06T21:43:14.057258+0000","last_unstale":"2024-02-06T22:17:53.359066+0000","last_undegraded":"2024-02-06T22:17:53.359066+0000","last_fullsized":"2024-02-06T22:17:53.359066+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T21:49:47.557758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":2582,"reported_epoch":1343,"state":"active+clean","last_fresh":"2024-02-06T22:17:54.416037+0000","last_change":"2024-02-06T21:42:49.763089+0000","last_active":"2024-02-06T22:17:54.416037+0000","last_peered":"2024-02-06T22:17:54.416037+0000","last_clean":"2024-02-06T22:17:54.416037+0000","last_became_active":"2024-02-06T21:42:49.754000+0000","last_became_peered":"2024-02-06T21:42:49.754000+0000","last_unstale":"2024-02-06T22:17:54.416037+0000","last_undegraded":"2024-02-06T22:17:54.416037+0000","last_fullsized":"2024-02-06T22:17:54.416037+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T06:23:56.755422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":18,"seq":77309411768,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72680,"kb_used_data":5404,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299160,"statfs":{"total":96636764160,"available":96562339840,"internally_reserved":0,"allocated":5533696,"data_stored":2787815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:41 2024","interfaces":[{"interface":"back","average":{"1min":15.664999999999999,"5min":29.364000000000001,"15min":24.5},"min":{"1min":5.9740000000000002,"5min":4.96,"15min":4.5209999999999999},"max":{"1min":76.638000000000005,"5min":223.274,"15min":549.32299999999998},"last":16.513000000000002},{"interface":"front","average":{"1min":20.262,"5min":35.253999999999998,"15min":29.021999999999998},"min":{"1min":10.795999999999999,"5min":7.9969999999999999,"15min":5.4900000000000002},"max":{"1min":78.369,"5min":224.31700000000001,"15min":368.31200000000001},"last":22.681999999999999}]},{"osd":2,"last update":"Tue Feb 6 22:17:09 2024","interfaces":[{"interface":"back","average":{"1min":23.908999999999999,"5min":30.920999999999999,"15min":26.542999999999999},"min":{"1min":11.26,"5min":5.1520000000000001,"15min":5.1520000000000001},"max":{"1min":88.763000000000005,"5min":205.28800000000001,"15min":387.75299999999999},"last":21.605},{"interface":"front","average":{"1min":12.706,"5min":29.155000000000001,"15min":31.622},"min":{"1min":5.3460000000000001,"5min":5.3460000000000001,"15min":5.3460000000000001},"max":{"1min":43.438000000000002,"5min":220.31899999999999,"15min":442.25},"last":5.492}]},{"osd":3,"last update":"Tue Feb 6 22:17:55 2024","interfaces":[{"interface":"back","average":{"1min":11.712999999999999,"5min":24.257999999999999,"15min":21.895},"min":{"1min":6.1520000000000001,"5min":5.3899999999999997,"15min":5.3899999999999997},"max":{"1min":68.462999999999994,"5min":359.19299999999998,"15min":359.19299999999998},"last":10.077},{"interface":"front","average":{"1min":26.09,"5min":35.533999999999999,"15min":31.292999999999999},"min":{"1min":11.449,"5min":8.5500000000000007,"15min":7.8010000000000002},"max":{"1min":102.22199999999999,"5min":266.40600000000001,"15min":462.86200000000002},"last":24.975999999999999}]},{"osd":4,"last update":"Tue Feb 6 22:17:24 2024","interfaces":[{"interface":"back","average":{"1min":25.393999999999998,"5min":39.593000000000004,"15min":29.221},"min":{"1min":12.670999999999999,"5min":9.0559999999999992,"15min":5.9539999999999997},"max":{"1min":85.082999999999998,"5min":535.22500000000002,"15min":535.22500000000002},"last":23.643999999999998},{"interface":"front","average":{"1min":12.597,"5min":28.677,"15min":22.689},"min":{"1min":7.5220000000000002,"5min":6.3010000000000002,"15min":5.4649999999999999},"max":{"1min":49.466000000000001,"5min":473.69799999999998,"15min":473.69799999999998},"last":8.5939999999999994}]},{"osd":5,"last update":"Tue Feb 6 22:17:41 2024","interfaces":[{"interface":"back","average":{"1min":27.221,"5min":36.164999999999999,"15min":32.499000000000002},"min":{"1min":14.074999999999999,"5min":8.0459999999999994,"15min":6.4550000000000001},"max":{"1min":90.605999999999995,"5min":260.74400000000003,"15min":454.49099999999999},"last":26.067},{"interface":"front","average":{"1min":17.254000000000001,"5min":25.736999999999998,"15min":26.802},"min":{"1min":8.1069999999999993,"5min":6.8010000000000002,"15min":6.109},"max":{"1min":70.180000000000007,"5min":147.011,"15min":539.04700000000003},"last":12.048}]},{"osd":6,"last update":"Tue Feb 6 22:17:24 2024","interfaces":[{"interface":"back","average":{"1min":27.564,"5min":40.116999999999997,"15min":37.177},"min":{"1min":13.347,"5min":12.289,"15min":10.353999999999999},"max":{"1min":89.951999999999998,"5min":268.73399999999998,"15min":482.846},"last":27.062999999999999},{"interface":"front","average":{"1min":16.326000000000001,"5min":27.145,"15min":24.978000000000002},"min":{"1min":8.7759999999999998,"5min":6.9349999999999996,"15min":6.9349999999999996},"max":{"1min":51.412999999999997,"5min":186.95500000000001,"15min":322.64999999999998},"last":15.586}]},{"osd":7,"last update":"Tue Feb 6 22:17:41 2024","interfaces":[{"interface":"back","average":{"1min":15.381,"5min":27.943000000000001,"15min":30.135000000000002},"min":{"1min":6.1669999999999998,"5min":6.1669999999999998,"15min":6.1669999999999998},"max":{"1min":75.899000000000001,"5min":249.69200000000001,"15min":452.22399999999999},"last":14.262},{"interface":"front","average":{"1min":31.131,"5min":42.412999999999997,"15min":35.392000000000003},"min":{"1min":13.923999999999999,"5min":10.964,"15min":7.7030000000000003},"max":{"1min":99.406000000000006,"5min":292.94600000000003,"15min":563.26900000000001},"last":28.564}]}]},{"osd":6,"up_from":17,"seq":73014444472,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72248,"kb_used_data":4972,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299592,"statfs":{"total":96636764160,"available":96562782208,"internally_reserved":0,"allocated":5091328,"data_stored":2367949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":14.135,"5min":21.481000000000002,"15min":17.268000000000001},"min":{"1min":4.3339999999999996,"5min":4.3230000000000004,"15min":4.3230000000000004},"max":{"1min":59.670999999999999,"5min":188.761,"15min":218.85400000000001},"last":13.481999999999999},{"interface":"front","average":{"1min":11.811,"5min":24.353000000000002,"15min":17.314},"min":{"1min":5.1559999999999997,"5min":5.1559999999999997,"15min":4.8449999999999998},"max":{"1min":38.082999999999998,"5min":193.68000000000001,"15min":193.68000000000001},"last":7.5709999999999997}]},{"osd":1,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":15.811999999999999,"5min":23.908999999999999,"15min":21.940000000000001},"min":{"1min":6.431,"5min":6.1459999999999999,"15min":5.109},"max":{"1min":54.000999999999998,"5min":180.447,"15min":204.64500000000001},"last":12.768000000000001},{"interface":"front","average":{"1min":17.199000000000002,"5min":23.283000000000001,"15min":24.012},"min":{"1min":6.7949999999999999,"5min":6.7949999999999999,"15min":6.1070000000000002},"max":{"1min":60.319000000000003,"5min":168.404,"15min":460.71600000000001},"last":8.9610000000000003}]},{"osd":2,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":19.161000000000001,"5min":36.042000000000002,"15min":26.562999999999999},"min":{"1min":8.0570000000000004,"5min":6.9359999999999999,"15min":5.5330000000000004},"max":{"1min":72.200000000000003,"5min":794.04200000000003,"15min":794.04200000000003},"last":11.016},{"interface":"front","average":{"1min":19.587,"5min":43.002000000000002,"15min":27.474},"min":{"1min":11.856,"5min":6.2999999999999998,"15min":5.601},"max":{"1min":62.237000000000002,"5min":795.17499999999995,"15min":795.17499999999995},"last":15.772}]},{"osd":3,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":19.741,"5min":32.057000000000002,"15min":24.117000000000001},"min":{"1min":10.901999999999999,"5min":7.718,"15min":5.9240000000000004},"max":{"1min":72.951999999999998,"5min":317.899,"15min":317.899},"last":11.653},{"interface":"front","average":{"1min":20.227,"5min":29.085999999999999,"15min":21.300999999999998},"min":{"1min":10.553000000000001,"5min":6.468,"15min":6.0380000000000003},"max":{"1min":65.974000000000004,"5min":405.108,"15min":405.108},"last":16.552}]},{"osd":4,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":10.769,"5min":23.568999999999999,"15min":22.443999999999999},"min":{"1min":5.8940000000000001,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":30.241,"5min":231.55699999999999,"15min":243.667},"last":6.4859999999999998},{"interface":"front","average":{"1min":14.613,"5min":30.960000000000001,"15min":27.713000000000001},"min":{"1min":8.641,"5min":7.7160000000000002,"15min":7.3399999999999999},"max":{"1min":39.484999999999999,"5min":284.37,"15min":339.34399999999999},"last":9.9689999999999994}]},{"osd":5,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":20.939,"5min":24.574999999999999,"15min":22.995000000000001},"min":{"1min":11.087999999999999,"5min":9.2929999999999993,"15min":7.5990000000000002},"max":{"1min":67.230000000000004,"5min":157.095,"15min":240.20500000000001},"last":14.972},{"interface":"front","average":{"1min":23.254999999999999,"5min":31.707999999999998,"15min":27.795000000000002},"min":{"1min":14.375,"5min":13.481,"15min":9.9260000000000002},"max":{"1min":69.085999999999999,"5min":202.673,"15min":223.42500000000001},"last":18.164999999999999}]},{"osd":7,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":23.655999999999999,"5min":30.402999999999999,"15min":23.09},"min":{"1min":15.859,"5min":11.272,"15min":7.4580000000000002},"max":{"1min":70.263999999999996,"5min":190.77799999999999,"15min":224.63},"last":17.344000000000001},{"interface":"front","average":{"1min":21.408999999999999,"5min":31.376999999999999,"15min":27.039999999999999},"min":{"1min":8.7430000000000003,"5min":8.7430000000000003,"15min":8.7430000000000003},"max":{"1min":74.259,"5min":203.26499999999999,"15min":220.893},"last":12.231}]}]},{"osd":5,"up_from":17,"seq":73014444475,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72664,"kb_used_data":5388,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299176,"statfs":{"total":96636764160,"available":96562356224,"internally_reserved":0,"allocated":5517312,"data_stored":2780247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":14.396000000000001,"5min":19.771999999999998,"15min":16.498999999999999},"min":{"1min":6.2709999999999999,"5min":5.6180000000000003,"15min":4.5090000000000003},"max":{"1min":72.215999999999994,"5min":115.977,"15min":350.07600000000002},"last":8.0860000000000003},{"interface":"front","average":{"1min":20.567,"5min":26.533999999999999,"15min":22.433},"min":{"1min":9.1340000000000003,"5min":7.8810000000000002,"15min":6.133},"max":{"1min":81.966999999999999,"5min":128.446,"15min":351.94},"last":22.359000000000002}]},{"osd":1,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":16.960000000000001,"5min":22.443000000000001,"15min":21.648},"min":{"1min":7.9630000000000001,"5min":7.0830000000000002,"15min":6.2359999999999998},"max":{"1min":46.289000000000001,"5min":158.76400000000001,"15min":275.12},"last":21.361000000000001},{"interface":"front","average":{"1min":19.393000000000001,"5min":24.099,"15min":22.812999999999999},"min":{"1min":8.5939999999999994,"5min":8.1590000000000007,"15min":4.6520000000000001},"max":{"1min":61.463000000000001,"5min":112.78700000000001,"15min":369.73200000000003},"last":24.346}]},{"osd":2,"last update":"Tue Feb 6 22:17:21 2024","interfaces":[{"interface":"back","average":{"1min":12.752000000000001,"5min":43.362000000000002,"15min":31.550000000000001},"min":{"1min":5.7140000000000004,"5min":5.7140000000000004,"15min":5.7140000000000004},"max":{"1min":46.295000000000002,"5min":2330.5610000000001,"15min":2330.5610000000001},"last":7.3239999999999998},{"interface":"front","average":{"1min":17.791,"5min":50.844999999999999,"15min":35.262999999999998},"min":{"1min":7.3230000000000004,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":44.875999999999998,"5min":2614.4409999999998,"15min":2614.4409999999998},"last":10.337}]},{"osd":3,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":18.696000000000002,"5min":23.902000000000001,"15min":24.218},"min":{"1min":7.8049999999999997,"5min":6.125,"15min":6.125},"max":{"1min":62.171999999999997,"5min":156.80000000000001,"15min":393.55200000000002},"last":11.554},{"interface":"front","average":{"1min":14.869,"5min":19.66,"15min":21.309999999999999},"min":{"1min":6.1159999999999997,"5min":5.3819999999999997,"15min":5.3819999999999997},"max":{"1min":63.813000000000002,"5min":104.755,"15min":376.113},"last":9.1809999999999992}]},{"osd":4,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":21.885000000000002,"15min":23.100999999999999},"min":{"1min":6.5650000000000004,"5min":6.093,"15min":6.093},"max":{"1min":84.843999999999994,"5min":105.532,"15min":426.77300000000002},"last":26.968},{"interface":"front","average":{"1min":18.773,"5min":21.994,"15min":24.087},"min":{"1min":7.4870000000000001,"5min":7.3819999999999997,"15min":5.8330000000000002},"max":{"1min":50.149000000000001,"5min":90.284000000000006,"15min":419.16000000000003},"last":19.175999999999998}]},{"osd":6,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":20.283000000000001,"5min":29.465,"15min":28.431999999999999},"min":{"1min":6.9560000000000004,"5min":6.9560000000000004,"15min":6.9560000000000004},"max":{"1min":85.828000000000003,"5min":219.14099999999999,"15min":392.71600000000001},"last":27.768999999999998},{"interface":"front","average":{"1min":21.872,"5min":25.550000000000001,"15min":26.341000000000001},"min":{"1min":9.8740000000000006,"5min":7.2350000000000003,"15min":7.2350000000000003},"max":{"1min":65.367000000000004,"5min":152.69900000000001,"15min":377.31400000000002},"last":19.942}]},{"osd":7,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":19.959,"5min":25.904,"15min":25.827999999999999},"min":{"1min":9.0510000000000002,"5min":9.0510000000000002,"15min":9.0510000000000002},"max":{"1min":63.646000000000001,"5min":124.711,"15min":377.87400000000002},"last":26.169},{"interface":"front","average":{"1min":19.725000000000001,"5min":27.969999999999999,"15min":24.045999999999999},"min":{"1min":7.8940000000000001,"5min":7.8940000000000001,"15min":6.7549999999999999},"max":{"1min":48.325000000000003,"5min":152.59399999999999,"15min":338.803},"last":28.356999999999999}]}]},{"osd":4,"up_from":17,"seq":73014444471,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72244,"kb_used_data":4968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299596,"statfs":{"total":96636764160,"available":96562786304,"internally_reserved":0,"allocated":5087232,"data_stored":2367944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.2439999999999998,"5min":17.853000000000002,"15min":16.888999999999999},"min":{"1min":4.907,"5min":4.4169999999999998,"15min":4.3899999999999997},"max":{"1min":19.225999999999999,"5min":167.291,"15min":216.86600000000001},"last":7.6379999999999999},{"interface":"front","average":{"1min":9.9450000000000003,"5min":22.707999999999998,"15min":18.631},"min":{"1min":4.742,"5min":4.6920000000000002,"15min":4.6829999999999998},"max":{"1min":23.745000000000001,"5min":231.82599999999999,"15min":287.97300000000001},"last":14.872}]},{"osd":1,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":17.431999999999999,"5min":24.254999999999999,"15min":30.890000000000001},"min":{"1min":6.2949999999999999,"5min":6.2949999999999999,"15min":5.1539999999999999},"max":{"1min":65.745999999999995,"5min":207.22399999999999,"15min":678.16999999999996},"last":9.7309999999999999},{"interface":"front","average":{"1min":15.343999999999999,"5min":23.465,"15min":32.118000000000002},"min":{"1min":8.4299999999999997,"5min":6.2599999999999998,"15min":6.2599999999999998},"max":{"1min":25.268999999999998,"5min":203.19999999999999,"15min":667.16300000000001},"last":11.487}]},{"osd":2,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":15.932,"5min":59.622999999999998,"15min":35.741999999999997},"min":{"1min":10.554,"5min":6.601,"15min":6.601},"max":{"1min":32.744999999999997,"5min":2277.7020000000002,"15min":2277.7020000000002},"last":15.648999999999999},{"interface":"front","average":{"1min":22.949999999999999,"5min":66.596999999999994,"15min":44.031999999999996},"min":{"1min":12.722,"5min":9.4559999999999995,"15min":9.4559999999999995},"max":{"1min":52.287999999999997,"5min":2279.0419999999999,"15min":2279.0419999999999},"last":17.969000000000001}]},{"osd":3,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":21.295999999999999,"5min":30.468,"15min":28.460999999999999},"min":{"1min":9.4499999999999993,"5min":6.3559999999999999,"15min":6.3559999999999999},"max":{"1min":62.805999999999997,"5min":254.292,"15min":423.20400000000001},"last":17.077999999999999},{"interface":"front","average":{"1min":15.101000000000001,"5min":31.655000000000001,"15min":24.419},"min":{"1min":6.9610000000000003,"5min":6.9610000000000003,"15min":5.5910000000000002},"max":{"1min":25.286999999999999,"5min":225.64699999999999,"15min":359.959},"last":12.189}]},{"osd":5,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":22.587,"5min":34.43,"15min":29.388999999999999},"min":{"1min":12.337999999999999,"5min":10.756,"15min":7.2999999999999998},"max":{"1min":66.715999999999994,"5min":248.70500000000001,"15min":363.36900000000003},"last":20.989000000000001},{"interface":"front","average":{"1min":15.039999999999999,"5min":28.181999999999999,"15min":27.018999999999998},"min":{"1min":6.6040000000000001,"5min":6.476,"15min":6.1550000000000002},"max":{"1min":43.223999999999997,"5min":208.02699999999999,"15min":809.58100000000002},"last":10.851000000000001}]},{"osd":6,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":17.777999999999999,"5min":24.445,"15min":26.818999999999999},"min":{"1min":7.5819999999999999,"5min":6.2930000000000001,"15min":6.2930000000000001},"max":{"1min":63.929000000000002,"5min":174.40700000000001,"15min":400.46800000000002},"last":12.803000000000001},{"interface":"front","average":{"1min":24.074999999999999,"5min":35.155000000000001,"15min":36.561},"min":{"1min":13.917,"5min":11.411,"15min":10.502000000000001},"max":{"1min":66.519999999999996,"5min":269.44200000000001,"15min":498.22300000000001},"last":20.058}]},{"osd":7,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":24.492000000000001,"5min":35.420000000000002,"15min":26.943000000000001},"min":{"1min":14.81,"5min":8.9009999999999998,"15min":6.6239999999999997},"max":{"1min":52.576999999999998,"5min":287.61599999999999,"15min":360.55599999999998},"last":18.937999999999999},{"interface":"front","average":{"1min":15.82,"5min":26.690999999999999,"15min":28.559999999999999},"min":{"1min":9.7210000000000001,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":28.702999999999999,"5min":215.26400000000001,"15min":478.649},"last":8.8460000000000001}]}]},{"osd":3,"up_from":17,"seq":73014444471,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72244,"kb_used_data":4968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299596,"statfs":{"total":96636764160,"available":96562786304,"internally_reserved":0,"allocated":5087232,"data_stored":2367944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:35 2024","interfaces":[{"interface":"back","average":{"1min":15.148999999999999,"5min":19.969999999999999,"15min":18.234000000000002},"min":{"1min":5.6100000000000003,"5min":4.5270000000000001,"15min":4.173},"max":{"1min":37.642000000000003,"5min":178.14400000000001,"15min":233.84200000000001},"last":10.285},{"interface":"front","average":{"1min":17.673999999999999,"5min":22.867000000000001,"15min":22.109999999999999},"min":{"1min":7.1299999999999999,"5min":5.3949999999999996,"15min":5.3949999999999996},"max":{"1min":45.851999999999997,"5min":224.822,"15min":224.822},"last":11.699999999999999}]},{"osd":1,"last update":"Tue Feb 6 22:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.183999999999999,"5min":19.210000000000001,"15min":21.574000000000002},"min":{"1min":6.7039999999999997,"5min":6.7039999999999997,"15min":6.7039999999999997},"max":{"1min":21.882999999999999,"5min":169.31100000000001,"15min":273.12700000000001},"last":14.042999999999999},{"interface":"front","average":{"1min":16.084,"5min":25.670000000000002,"15min":25.431999999999999},"min":{"1min":10.845000000000001,"5min":10.845000000000001,"15min":9.2880000000000003},"max":{"1min":24.265000000000001,"5min":205.495,"15min":322.70499999999998},"last":12.353}]},{"osd":2,"last update":"Tue Feb 6 22:17:35 2024","interfaces":[{"interface":"back","average":{"1min":25.116,"5min":28.814,"15min":26.242999999999999},"min":{"1min":11.129,"5min":11.129,"15min":9.1999999999999993},"max":{"1min":59.799999999999997,"5min":240.316,"15min":282.40699999999998},"last":13.276},{"interface":"front","average":{"1min":11.965,"5min":28.228999999999999,"15min":21.456},"min":{"1min":5.8979999999999997,"5min":5.8419999999999996,"15min":5.1550000000000002},"max":{"1min":39.423000000000002,"5min":998.23500000000001,"15min":998.23500000000001},"last":6.3099999999999996}]},{"osd":4,"last update":"Tue Feb 6 22:17:17 2024","interfaces":[{"interface":"back","average":{"1min":12.571999999999999,"5min":17.777000000000001,"15min":20.849},"min":{"1min":7.6459999999999999,"5min":6.1109999999999998,"15min":5.8339999999999996},"max":{"1min":25.420000000000002,"5min":162.65299999999999,"15min":189.12},"last":8.266},{"interface":"front","average":{"1min":18.823,"5min":27.978999999999999,"15min":31.401},"min":{"1min":12.393000000000001,"5min":11.710000000000001,"15min":11.045},"max":{"1min":30.169,"5min":176.35400000000001,"15min":235.315},"last":15.599}]},{"osd":5,"last update":"Tue Feb 6 22:17:35 2024","interfaces":[{"interface":"back","average":{"1min":29.649000000000001,"5min":32.82,"15min":29.718},"min":{"1min":10.526,"5min":10.087999999999999,"15min":8.9920000000000009},"max":{"1min":77.253,"5min":264.06999999999999,"15min":626.27700000000004},"last":14.845000000000001},{"interface":"front","average":{"1min":37.348999999999997,"5min":37.712000000000003,"15min":32.811999999999998},"min":{"1min":13.699,"5min":13.699,"15min":13.699},"max":{"1min":112.84699999999999,"5min":284.88299999999998,"15min":614.85400000000004},"last":16.399000000000001}]},{"osd":6,"last update":"Tue Feb 6 22:16:52 2024","interfaces":[{"interface":"back","average":{"1min":22.713999999999999,"5min":24.713999999999999,"15min":26.106999999999999},"min":{"1min":8.7240000000000002,"5min":6.4550000000000001,"15min":6.218},"max":{"1min":55.887,"5min":177.40700000000001,"15min":954.18200000000002},"last":17.332000000000001},{"interface":"front","average":{"1min":16.166,"5min":29.777000000000001,"15min":35.177},"min":{"1min":8.3729999999999993,"5min":8.3729999999999993,"15min":8.3729999999999993},"max":{"1min":40.037999999999997,"5min":209.25800000000001,"15min":954.79300000000001},"last":7.5289999999999999}]},{"osd":7,"last update":"Tue Feb 6 22:16:52 2024","interfaces":[{"interface":"back","average":{"1min":15.278,"5min":21.925000000000001,"15min":25.556999999999999},"min":{"1min":8.2970000000000006,"5min":7.3579999999999997,"15min":7.319},"max":{"1min":40.340000000000003,"5min":173.642,"15min":655.048},"last":9.3759999999999994},{"interface":"front","average":{"1min":27.821999999999999,"5min":31.497,"15min":33.704999999999998},"min":{"1min":16.033000000000001,"5min":14.802,"15min":6.9199999999999999},"max":{"1min":69.661000000000001,"5min":206.87,"15min":653.67600000000004},"last":18.169}]}]},{"osd":2,"up_from":17,"seq":73014444471,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72244,"kb_used_data":4968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299596,"statfs":{"total":96636764160,"available":96562786304,"internally_reserved":0,"allocated":5087232,"data_stored":2367944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":9.8650000000000002,"5min":40.578000000000003,"15min":27.196999999999999},"min":{"1min":5.3460000000000001,"5min":4.6050000000000004,"15min":4.3150000000000004},"max":{"1min":23.093,"5min":1631.963,"15min":1631.963},"last":13.194000000000001},{"interface":"front","average":{"1min":9.0060000000000002,"5min":21.448,"15min":21.940999999999999},"min":{"1min":4.4859999999999998,"5min":4.4859999999999998,"15min":3.9630000000000001},"max":{"1min":17.125,"5min":138.09800000000001,"15min":370.68799999999999},"last":8.4320000000000004}]},{"osd":1,"last update":"Tue Feb 6 22:17:26 2024","interfaces":[{"interface":"back","average":{"1min":10.836,"5min":52.090000000000003,"15min":32.5},"min":{"1min":6.9000000000000004,"5min":5.9340000000000002,"15min":4.492},"max":{"1min":38.810000000000002,"5min":3146.2530000000002,"15min":3146.2530000000002},"last":6.0919999999999996},{"interface":"front","average":{"1min":19.111000000000001,"5min":58.341999999999999,"15min":36.834000000000003},"min":{"1min":8.4320000000000004,"5min":6.9669999999999996,"15min":6.9020000000000001},"max":{"1min":111.504,"5min":3147.6979999999999,"15min":3147.6979999999999},"last":9.0259999999999998}]},{"osd":3,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":11.875999999999999,"5min":51.591999999999999,"15min":33.267000000000003},"min":{"1min":8.4879999999999995,"5min":7.3890000000000002,"15min":6.1239999999999997},"max":{"1min":19.702999999999999,"5min":2357.8649999999998,"15min":2357.8649999999998},"last":7.7939999999999996},{"interface":"front","average":{"1min":17.119,"5min":58.719000000000001,"15min":37.293999999999997},"min":{"1min":9.6289999999999996,"5min":9.6289999999999996,"15min":7.133},"max":{"1min":27.391999999999999,"5min":2367.828,"15min":2367.828},"last":15.157}]},{"osd":4,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":12.506,"5min":45.119,"15min":36.264000000000003},"min":{"1min":7.5949999999999998,"5min":6.5129999999999999,"15min":6.5129999999999999},"max":{"1min":17.905000000000001,"5min":1805.6099999999999,"15min":1805.6099999999999},"last":10.436999999999999},{"interface":"front","average":{"1min":15.993,"5min":22.584,"15min":26.597999999999999},"min":{"1min":10.06,"5min":6.976,"15min":6.4009999999999998},"max":{"1min":26.065999999999999,"5min":139.29900000000001,"15min":366.25099999999998},"last":11.781000000000001}]},{"osd":5,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":13.83,"5min":59.646000000000001,"15min":39.192},"min":{"1min":8.7680000000000007,"5min":7.0650000000000004,"15min":6.9610000000000003},"max":{"1min":21.802,"5min":3145.4110000000001,"15min":3145.4110000000001},"last":11.118},{"interface":"front","average":{"1min":18.478000000000002,"5min":58.215000000000003,"15min":41.988999999999997},"min":{"1min":12.645,"5min":8.3140000000000001,"15min":8.3140000000000001},"max":{"1min":28.748000000000001,"5min":2302.047,"15min":2302.047},"last":14.541}]},{"osd":6,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":13.406000000000001,"5min":31.114000000000001,"15min":30.577999999999999},"min":{"1min":10.297000000000001,"5min":7.0819999999999999,"15min":6.6479999999999997},"max":{"1min":19.422999999999998,"5min":266.53500000000003,"15min":369.185},"last":12.52},{"interface":"front","average":{"1min":19.795000000000002,"5min":60.094000000000001,"15min":42.265999999999998},"min":{"1min":14.802,"5min":9.4570000000000007,"15min":8.4930000000000003},"max":{"1min":30.856000000000002,"5min":2302.8209999999999,"15min":2302.8209999999999},"last":17.099}]},{"osd":7,"last update":"Tue Feb 6 22:16:54 2024","interfaces":[{"interface":"back","average":{"1min":15.435,"5min":27.277000000000001,"15min":28.007999999999999},"min":{"1min":9.0050000000000008,"5min":8.5790000000000006,"15min":8.4619999999999997},"max":{"1min":23.422000000000001,"5min":267.28300000000002,"15min":405.29899999999998},"last":13.826000000000001},{"interface":"front","average":{"1min":17.768999999999998,"5min":31.587,"15min":31.151},"min":{"1min":12.055,"5min":11.515000000000001,"15min":10.06},"max":{"1min":25.113,"5min":268.60899999999998,"15min":342.22699999999998},"last":18.021999999999998}]}]},{"osd":7,"up_from":13,"seq":55834575287,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76432,"kb_used_data":4996,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295408,"statfs":{"total":96636764160,"available":96558497792,"internally_reserved":0,"allocated":5115904,"data_stored":2382407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":11.686999999999999,"5min":23.573,"15min":18.436},"min":{"1min":5.0789999999999997,"5min":4.2880000000000003,"15min":4.2880000000000003},"max":{"1min":28.937000000000001,"5min":283.42899999999997,"15min":283.42899999999997},"last":8.8049999999999997},{"interface":"front","average":{"1min":12.000999999999999,"5min":23.219999999999999,"15min":19.027999999999999},"min":{"1min":5.2949999999999999,"5min":4.3730000000000002,"15min":4.3730000000000002},"max":{"1min":33.189,"5min":279.459,"15min":387.32999999999998},"last":10.821999999999999}]},{"osd":1,"last update":"Tue Feb 6 22:17:29 2024","interfaces":[{"interface":"back","average":{"1min":13.608000000000001,"5min":19.052,"15min":22.741},"min":{"1min":6.0609999999999999,"5min":6.0609999999999999,"15min":5.577},"max":{"1min":31.280999999999999,"5min":145.70599999999999,"15min":325.41699999999997},"last":11.997999999999999},{"interface":"front","average":{"1min":13.199999999999999,"5min":17.952000000000002,"15min":21.225999999999999},"min":{"1min":6.2720000000000002,"5min":6.2720000000000002,"15min":4.7960000000000003},"max":{"1min":30.079999999999998,"5min":159.56100000000001,"15min":380.70800000000003},"last":15.004}]},{"osd":2,"last update":"Tue Feb 6 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":15.321999999999999,"5min":78.852000000000004,"15min":42.732999999999997},"min":{"1min":7.1890000000000001,"5min":6.367,"15min":5.8769999999999998},"max":{"1min":56.795000000000002,"5min":3537.9119999999998,"15min":3537.9119999999998},"last":9.5879999999999992},{"interface":"front","average":{"1min":17,"5min":81.25,"15min":44.497},"min":{"1min":9.3260000000000005,"5min":7.6790000000000003,"15min":6.6539999999999999},"max":{"1min":26.536999999999999,"5min":3538.9259999999999,"15min":3538.9259999999999},"last":18.199000000000002}]},{"osd":3,"last update":"Tue Feb 6 22:17:29 2024","interfaces":[{"interface":"back","average":{"1min":16.347999999999999,"5min":20.353999999999999,"15min":21.109999999999999},"min":{"1min":7.7990000000000004,"5min":6.8399999999999999,"15min":6.3390000000000004},"max":{"1min":57.917999999999999,"5min":186.297,"15min":317.94600000000003},"last":12.984999999999999},{"interface":"front","average":{"1min":22.106000000000002,"5min":27.684000000000001,"15min":27.027999999999999},"min":{"1min":14.65,"5min":9.8360000000000003,"15min":9.8360000000000003},"max":{"1min":58.795000000000002,"5min":201.75299999999999,"15min":325.60599999999999},"last":14.130000000000001}]},{"osd":4,"last update":"Tue Feb 6 22:17:20 2024","interfaces":[{"interface":"back","average":{"1min":20.239999999999998,"5min":22.088999999999999,"15min":22.254999999999999},"min":{"1min":9.7159999999999993,"5min":9.2370000000000001,"15min":6.9690000000000003},"max":{"1min":61.972999999999999,"5min":102.001,"15min":235.27799999999999},"last":16.350999999999999},{"interface":"front","average":{"1min":14.773999999999999,"5min":18.849,"15min":25.335000000000001},"min":{"1min":7.585,"5min":6.8440000000000003,"15min":6.8440000000000003},"max":{"1min":25.353000000000002,"5min":87.968999999999994,"15min":350.33300000000003},"last":8.2149999999999999}]},{"osd":5,"last update":"Tue Feb 6 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":24.497,"5min":25.561,"15min":24.407},"min":{"1min":12.058,"5min":7.1189999999999998,"15min":7.1189999999999998},"max":{"1min":89.010999999999996,"5min":201.70400000000001,"15min":214.83500000000001},"last":19.277000000000001},{"interface":"front","average":{"1min":18.481999999999999,"5min":22.195,"15min":25.890999999999998},"min":{"1min":8.6969999999999992,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":65.813999999999993,"5min":106.07299999999999,"15min":301.78399999999999},"last":15.667}]},{"osd":6,"last update":"Tue Feb 6 22:17:29 2024","interfaces":[{"interface":"back","average":{"1min":23.632999999999999,"5min":26.312999999999999,"15min":25.606000000000002},"min":{"1min":15.241,"5min":8.3239999999999998,"15min":8.3239999999999998},"max":{"1min":60.616,"5min":203.58199999999999,"15min":235.27799999999999},"last":17.545999999999999},{"interface":"front","average":{"1min":16.300999999999998,"5min":22.77,"15min":26.331},"min":{"1min":9.3000000000000007,"5min":7.3179999999999996,"15min":6.7400000000000002},"max":{"1min":28.381,"5min":185.49799999999999,"15min":490.68299999999999},"last":10.222}]}]},{"osd":0,"up_from":13,"seq":55834575284,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72260,"kb_used_data":4984,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299580,"statfs":{"total":96636764160,"available":96562769920,"internally_reserved":0,"allocated":5103616,"data_stored":2375664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 6 22:17:53 2024","interfaces":[{"interface":"back","average":{"1min":15.26,"5min":28.606999999999999,"15min":23.428000000000001},"min":{"1min":5.6970000000000001,"5min":4.633,"15min":4.4029999999999996},"max":{"1min":98.292000000000002,"5min":314.11000000000001,"15min":416.55599999999998},"last":13.718},{"interface":"front","average":{"1min":13.664999999999999,"5min":22.065999999999999,"15min":21.591000000000001},"min":{"1min":6.5339999999999998,"5min":5.5389999999999997,"15min":4.5},"max":{"1min":81.653999999999996,"5min":199.238,"15min":407.44799999999998},"last":6.7539999999999996}]},{"osd":2,"last update":"Tue Feb 6 22:17:53 2024","interfaces":[{"interface":"back","average":{"1min":17.803999999999998,"5min":28.433,"15min":27.036999999999999},"min":{"1min":7.2770000000000001,"5min":6.4880000000000004,"15min":6.3799999999999999},"max":{"1min":93.007999999999996,"5min":264.48200000000003,"15min":395.14800000000002},"last":8.7899999999999991},{"interface":"front","average":{"1min":15.856,"5min":29.306000000000001,"15min":25.323},"min":{"1min":5.9100000000000001,"5min":5.5259999999999998,"15min":5.383},"max":{"1min":99.483000000000004,"5min":514.94500000000005,"15min":514.94500000000005},"last":7.6379999999999999}]},{"osd":3,"last update":"Tue Feb 6 22:17:53 2024","interfaces":[{"interface":"back","average":{"1min":19.762,"5min":26.550000000000001,"15min":24.818000000000001},"min":{"1min":8.6289999999999996,"5min":7.2409999999999997,"15min":5.8140000000000001},"max":{"1min":105.68000000000001,"5min":291.59100000000001,"15min":376.95100000000002},"last":11.481},{"interface":"front","average":{"1min":14.304,"5min":25.513000000000002,"15min":25.059999999999999},"min":{"1min":7.75,"5min":6.6139999999999999,"15min":6.6139999999999999},"max":{"1min":80.340000000000003,"5min":262.06900000000002,"15min":395.83100000000002},"last":10.786}]},{"osd":4,"last update":"Tue Feb 6 22:17:09 2024","interfaces":[{"interface":"back","average":{"1min":16.298999999999999,"5min":31.213000000000001,"15min":28.899000000000001},"min":{"1min":9.8650000000000002,"5min":7.3609999999999998,"15min":6.6970000000000001},"max":{"1min":27.573,"5min":491.53100000000001,"15min":491.53100000000001},"last":17.044},{"interface":"front","average":{"1min":11.919,"5min":26.989999999999998,"15min":25.725000000000001},"min":{"1min":8.4060000000000006,"5min":7.2409999999999997,"15min":6.6589999999999998},"max":{"1min":18.683,"5min":490.28699999999998,"15min":490.28699999999998},"last":9.6110000000000007}]},{"osd":5,"last update":"Tue Feb 6 22:17:53 2024","interfaces":[{"interface":"back","average":{"1min":22.463999999999999,"5min":31.809999999999999,"15min":28.338000000000001},"min":{"1min":10.941000000000001,"5min":7.9969999999999999,"15min":7.9969999999999999},"max":{"1min":104.276,"5min":282.00799999999998,"15min":480.24200000000002},"last":14.631},{"interface":"front","average":{"1min":22.041,"5min":27.257000000000001,"15min":23.969000000000001},"min":{"1min":9.7750000000000004,"5min":8.9350000000000005,"15min":7.6059999999999999},"max":{"1min":94.417000000000002,"5min":193.21700000000001,"15min":347.78699999999998},"last":13.004}]},{"osd":6,"last update":"Tue Feb 6 22:17:53 2024","interfaces":[{"interface":"back","average":{"1min":24.131,"5min":32.994999999999997,"15min":30.001999999999999},"min":{"1min":12.452,"5min":9.4789999999999992,"15min":8.9139999999999997},"max":{"1min":106.277,"5min":218.12,"15min":550.89700000000005},"last":16.227},{"interface":"front","average":{"1min":20.716000000000001,"5min":27.524999999999999,"15min":25.503},"min":{"1min":10.202,"5min":9.1310000000000002,"15min":8.4730000000000008},"max":{"1min":96.307000000000002,"5min":175.947,"15min":486.25299999999999},"last":10.202}]},{"osd":7,"last update":"Tue Feb 6 22:17:38 2024","interfaces":[{"interface":"back","average":{"1min":24.734000000000002,"5min":33.390999999999998,"15min":30.097000000000001},"min":{"1min":13.772,"5min":11.374000000000001,"15min":9.6170000000000009},"max":{"1min":107.01000000000001,"5min":218.108,"15min":404.22800000000001},"last":15.186999999999999},{"interface":"front","average":{"1min":28.641999999999999,"5min":38.777999999999999,"15min":34.070999999999998},"min":{"1min":15.483000000000001,"5min":13.800000000000001,"15min":11.497999999999999},"max":{"1min":109.504,"5min":219.56299999999999,"15min":415.92000000000002},"last":17.835999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-06T22:17:58.899 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-06T22:17:59.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.009+0000 7ffb1733d640 10 monclient: get_monmap_and_config 2024-02-06T22:17:59.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.009+0000 7ffb1733d640 10 monclient: build_initial_monmap 2024-02-06T22:17:59.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.009+0000 7ffb1733d640 10 monclient: monmap: 2024-02-06T22:17:59.010 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:17:59.011 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:59.011 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:17:59.010388+0000 2024-02-06T22:17:59.011 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:17:59.010388+0000 2024-02-06T22:17:59.011 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:17:59.011 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:59.011 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:17:59.011 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:17:59.011 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:17:59.011 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:17:59.011 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:17:59.011 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:59.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.009+0000 7ffb1733d640 1 Processor -- start 2024-02-06T22:17:59.013 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.009+0000 7ffb1733d640 1 -- start start 2024-02-06T22:17:59.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.009+0000 7ffb1733d640 10 monclient: init 2024-02-06T22:17:59.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.010+0000 7ffb1733d640 10 monclient: _reopen_session rank -1 2024-02-06T22:17:59.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.010+0000 7ffb1733d640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-02-06T22:17:59.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.010+0000 7ffb1733d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffb100b54b0 addr v1:172.21.15.120:6789/0 2024-02-06T22:17:59.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.010+0000 7ffb1733d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb1013ff40 0x7ffb100a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:59.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.010+0000 7ffb1733d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb1013ff40 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:17:59.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.010+0000 7ffb1733d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffb10142680 0x7ffb10142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:59.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.010+0000 7ffb1733d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb10142680 addr v2:172.21.15.120:3300/0 2024-02-06T22:17:59.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.010+0000 7ffb1733d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb100ab720 con 0x7ffb100b54b0 2024-02-06T22:17:59.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.010+0000 7ffb1733d640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:59.014 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.010+0000 7ffb1733d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb100ab520 con 0x7ffb10142680 2024-02-06T22:17:59.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.010+0000 7ffb1733d640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:59.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.010+0000 7ffb1733d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb100b5890 con 0x7ffb1013ff40 2024-02-06T22:17:59.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.010+0000 7ffb1733d640 10 monclient(hunting): _renew_subs 2024-02-06T22:17:59.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.010+0000 7ffb1733d640 10 monclient(hunting): authenticate will time out at 3509.348047s 2024-02-06T22:17:59.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.011+0000 7ffb148d8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb1013ff40 0x7ffb100a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:59.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.011+0000 7ffb158da640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffb10142680 0x7ffb10142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:59.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.011+0000 7ffb148d8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb1013ff40 0x7ffb100a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:58544/0 (socket says 172.21.15.120:58544) 2024-02-06T22:17:59.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.011+0000 7ffb148d8640 1 -- 172.21.15.120:0/2216359806 learned_addr learned my addr 172.21.15.120:0/2216359806 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:17:59.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.011+0000 7ffb148d8640 10 monclient(hunting): get_auth_request con 0x7ffb1013ff40 auth_method 0 2024-02-06T22:17:59.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.011+0000 7ffb148d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:59.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.011+0000 7ffb148d8640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:59.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.011+0000 7ffb148d8640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:59.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.012+0000 7ffb148d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:59.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.012+0000 7ffb148d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:59.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.012+0000 7ffb148d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:59.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.012+0000 7ffb158da640 10 monclient(hunting): get_auth_request con 0x7ffb10142680 auth_method 0 2024-02-06T22:17:59.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.012+0000 7ffb158da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:59.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.012+0000 7ffb158da640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:59.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.012+0000 7ffb158da640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:59.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.013+0000 7ffb158da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:59.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.013+0000 7ffb158da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:59.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.013+0000 7ffb158da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:59.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.013+0000 7ffb148d8640 10 monclient(hunting): handle_auth_done global_id 18686 payload 274 2024-02-06T22:17:59.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.013+0000 7ffb148d8640 1 -- 172.21.15.120:0/2216359806 >> v2:172.21.15.120:3300/0 conn(0x7ffb10142680 msgr2=0x7ffb10142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:17:59.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.013+0000 7ffb148d8640 1 --2- 172.21.15.120:0/2216359806 >> v2:172.21.15.120:3300/0 conn(0x7ffb10142680 0x7ffb10142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:59.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.013+0000 7ffb148d8640 1 -- 172.21.15.120:0/2216359806 >> v1:172.21.15.120:6789/0 conn(0x7ffb100b54b0 legacy=0x7ffb1013f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:59.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.013+0000 7ffb148d8640 10 monclient: _finish_hunting 0 2024-02-06T22:17:59.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.013+0000 7ffb148d8640 1 monclient: found mon.noname-b 2024-02-06T22:17:59.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.013+0000 7ffb148d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:17:59.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.013+0000 7ffb148d8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T22:17:59.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.013+0000 7ffb148d8640 1 -- 172.21.15.120:0/2216359806 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb100b4800 con 0x7ffb1013ff40 2024-02-06T22:17:59.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.015+0000 7ffb148d8640 1 --2- 172.21.15.120:0/2216359806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb1013ff40 0x7ffb100a9860 secure :-1 s=READY pgs=3340 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009870 tx=0x7ffb0c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=643ac9588e082c23 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:17:59.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb07fff640 1 -- 172.21.15.120:0/2216359806 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c030840 con 0x7ffb1013ff40 2024-02-06T22:17:59.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:59.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb07fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:17:59.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb07fff640 10 monclient: dump: 2024-02-06T22:17:59.021 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:59.021 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:59.021 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:59.021 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:59.021 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:59.021 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:59.022 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:59.022 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:59.022 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:59.022 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:59.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb07fff640 10 monclient: _finish_auth 0 2024-02-06T22:17:59.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb07fff640 10 monclient: _check_auth_tickets 2024-02-06T22:17:59.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:17:59.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb07fff640 1 -- 172.21.15.120:0/2216359806 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c0309a0 con 0x7ffb1013ff40 2024-02-06T22:17:59.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb07fff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:17:59.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb07fff640 1 -- 172.21.15.120:0/2216359806 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c030d30 con 0x7ffb1013ff40 2024-02-06T22:17:59.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb1733d640 5 monclient: authenticate success, global_id 18686 2024-02-06T22:17:59.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb1733d640 10 monclient: get_monmap_and_config success 2024-02-06T22:17:59.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb1733d640 4 set_mon_vals no callback set 2024-02-06T22:17:59.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:59.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T22:17:59.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.019+0000 7ffb07fff640 10 monclient: dump: 2024-02-06T22:17:59.023 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:59.023 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:59.023 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:59.023 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:59.023 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:59.023 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:59.023 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:59.023 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:59.024 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:59.024 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:59.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.020+0000 7ffb1733d640 10 monclient: shutdown 2024-02-06T22:17:59.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.020+0000 7ffb1733d640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:17:59.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.020+0000 7ffb1733d640 1 -- 172.21.15.120:0/2216359806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb1013ff40 msgr2=0x7ffb100a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:59.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.020+0000 7ffb1733d640 1 --2- 172.21.15.120:0/2216359806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb1013ff40 0x7ffb100a9860 secure :-1 s=READY pgs=3340 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009870 tx=0x7ffb0c02b980 comp rx=0 tx=0).stop 2024-02-06T22:17:59.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.020+0000 7ffb1733d640 1 -- 172.21.15.120:0/2216359806 shutdown_connections 2024-02-06T22:17:59.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.020+0000 7ffb1733d640 1 --2- 172.21.15.120:0/2216359806 >> v2:172.21.15.120:3300/0 conn(0x7ffb10142680 0x7ffb10142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:59.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.020+0000 7ffb1733d640 1 --2- 172.21.15.120:0/2216359806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb1013ff40 0x7ffb100a9860 unknown :-1 s=CLOSED pgs=3340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:59.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.020+0000 7ffb1733d640 1 -- 172.21.15.120:0/2216359806 >> 172.21.15.120:0/2216359806 conn(0x7ffb100b8660 msgr2=0x7ffb100b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:17:59.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.020+0000 7ffb1733d640 1 -- 172.21.15.120:0/2216359806 shutdown_connections 2024-02-06T22:17:59.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.020+0000 7ffb1733d640 1 -- 172.21.15.120:0/2216359806 wait complete. 2024-02-06T22:17:59.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.021+0000 7ffb1733d640 10 monclient: build_initial_monmap 2024-02-06T22:17:59.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.021+0000 7ffb1733d640 10 monclient: monmap: 2024-02-06T22:17:59.029 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:17:59.030 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:17:59.030 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:17:59.030 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:17:59.030 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:17:59.030 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:59.030 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:17:59.030 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:17:59.030 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:17:59.030 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:59.030 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.021+0000 7ffb1733d640 1 librados: starting msgr at 2024-02-06T22:17:59.030 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.021+0000 7ffb1733d640 1 librados: starting objecter 2024-02-06T22:17:59.030 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.021+0000 7ffb1733d640 1 Processor -- start 2024-02-06T22:17:59.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.021+0000 7ffb1733d640 1 -- start start 2024-02-06T22:17:59.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.021+0000 7ffb1733d640 1 librados: setting wanted keys 2024-02-06T22:17:59.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.021+0000 7ffb1733d640 1 librados: calling monclient init 2024-02-06T22:17:59.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.021+0000 7ffb1733d640 10 monclient: init 2024-02-06T22:17:59.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 10 monclient: _reopen_session rank -1 2024-02-06T22:17:59.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T22:17:59.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb100b54b0 0x7ffb101365b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:59.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb100b54b0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:17:59.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb1013ff40 0x7ffb101331f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:59.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb1013ff40 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:17:59.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb10142680 0x7ffb10136ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:59.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb10142680 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:17:59.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:59.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb10138f40 con 0x7ffb1013ff40 2024-02-06T22:17:59.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:59.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb101383d0 con 0x7ffb100b54b0 2024-02-06T22:17:59.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 10 monclient(hunting): start opening mon connection 2024-02-06T22:17:59.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb101386b0 con 0x7ffb10142680 2024-02-06T22:17:59.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 10 monclient(hunting): _renew_subs 2024-02-06T22:17:59.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb1733d640 10 monclient(hunting): authenticate will time out at 3509.359758s 2024-02-06T22:17:59.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb158da640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb10142680 0x7ffb10136ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:59.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb148d8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb1013ff40 0x7ffb101331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:59.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb150d9640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb100b54b0 0x7ffb101365b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:59.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb148d8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb1013ff40 0x7ffb101331f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:41532/0 (socket says 172.21.15.120:41532) 2024-02-06T22:17:59.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb148d8640 1 -- 172.21.15.120:0/3773961765 learned_addr learned my addr 172.21.15.120:0/3773961765 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:17:59.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb148d8640 10 monclient(hunting): get_auth_request con 0x7ffb1013ff40 auth_method 0 2024-02-06T22:17:59.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb148d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:59.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb148d8640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:59.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb148d8640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:59.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb158da640 10 monclient(hunting): get_auth_request con 0x7ffb10142680 auth_method 0 2024-02-06T22:17:59.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.022+0000 7ffb158da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:59.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.023+0000 7ffb158da640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:59.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.023+0000 7ffb158da640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:59.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.023+0000 7ffb150d9640 10 monclient(hunting): get_auth_request con 0x7ffb100b54b0 auth_method 0 2024-02-06T22:17:59.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.023+0000 7ffb150d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:17:59.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.023+0000 7ffb150d9640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:17:59.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.023+0000 7ffb150d9640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:17:59.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.023+0000 7ffb148d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:59.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.023+0000 7ffb148d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:59.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.023+0000 7ffb148d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:59.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.023+0000 7ffb158da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:59.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.023+0000 7ffb158da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:59.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.023+0000 7ffb158da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:59.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.023+0000 7ffb150d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:17:59.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.023+0000 7ffb150d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:17:59.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.023+0000 7ffb150d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:17:59.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.026+0000 7ffb158da640 10 monclient(hunting): handle_auth_done global_id 18689 payload 824 2024-02-06T22:17:59.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.026+0000 7ffb158da640 1 -- 172.21.15.120:0/3773961765 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb100b54b0 msgr2=0x7ffb101365b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:59.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.026+0000 7ffb158da640 1 --2- 172.21.15.120:0/3773961765 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb100b54b0 0x7ffb101365b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:59.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.026+0000 7ffb158da640 1 -- 172.21.15.120:0/3773961765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb1013ff40 msgr2=0x7ffb101331f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:59.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.026+0000 7ffb158da640 1 --2- 172.21.15.120:0/3773961765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb1013ff40 0x7ffb101331f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:59.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.026+0000 7ffb158da640 10 monclient: _finish_hunting 0 2024-02-06T22:17:59.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.026+0000 7ffb158da640 1 monclient: found mon.noname-c 2024-02-06T22:17:59.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.026+0000 7ffb158da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:17:59.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.026+0000 7ffb158da640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T22:17:59.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.026+0000 7ffb158da640 1 -- 172.21.15.120:0/3773961765 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c0094d0 con 0x7ffb10142680 2024-02-06T22:17:59.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.028+0000 7ffb158da640 1 --2- 172.21.15.120:0/3773961765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb10142680 0x7ffb10136ba0 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800eba0 tx=0x7ffb0800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:17:59.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.028+0000 7ffb05ffb640 10 client.?.objecter ms_handle_connect 0x7ffb10142680 2024-02-06T22:17:59.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.028+0000 7ffb05ffb640 10 client.?.objecter resend_mon_ops 2024-02-06T22:17:59.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb05ffb640 1 -- 172.21.15.120:0/3773961765 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0800cd20 con 0x7ffb10142680 2024-02-06T22:17:59.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:59.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb05ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T22:17:59.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb05ffb640 10 monclient: dump: 2024-02-06T22:17:59.037 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:59.037 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:59.037 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:59.037 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:59.038 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:59.038 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:59.038 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:59.038 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:59.038 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:59.038 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:59.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb05ffb640 10 monclient: _finish_auth 0 2024-02-06T22:17:59.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb05ffb640 10 monclient: _check_auth_tickets 2024-02-06T22:17:59.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:17:59.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb05ffb640 1 -- 172.21.15.120:0/3773961765 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0800f040 con 0x7ffb10142680 2024-02-06T22:17:59.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb05ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:17:59.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb05ffb640 1 -- 172.21.15.120:0/3773961765 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb080146c0 con 0x7ffb10142680 2024-02-06T22:17:59.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:17:59.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T22:17:59.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb05ffb640 10 monclient: dump: 2024-02-06T22:17:59.039 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:17:59.039 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:17:59.039 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:59.039 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:17:59.039 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:17:59.039 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:17:59.039 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:17:59.039 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:17:59.039 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:17:59.040 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:17:59.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb1733d640 5 monclient: authenticate success, global_id 18689 2024-02-06T22:17:59.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb1733d640 10 monclient: _renew_subs 2024-02-06T22:17:59.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb077fe640 4 set_mon_vals no callback set 2024-02-06T22:17:59.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb1733d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:17:59.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.031+0000 7ffb1733d640 1 -- 172.21.15.120:0/3773961765 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb10137400 con 0x7ffb10142680 2024-02-06T22:17:59.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.032+0000 7ffb1733d640 10 client.18689.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:17:59.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.032+0000 7ffb1733d640 10 monclient: _renew_subs 2024-02-06T22:17:59.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.032+0000 7ffb1733d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:17:59.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.032+0000 7ffb1733d640 1 -- 172.21.15.120:0/3773961765 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb10138b40 con 0x7ffb10142680 2024-02-06T22:17:59.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.032+0000 7ffb1733d640 1 librados: init done 2024-02-06T22:17:59.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.033+0000 7ffb1733d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T22:17:59.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.033+0000 7ffb1733d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:17:59.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.033+0000 7ffb1733d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:17:59.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.033+0000 7ffb1733d640 1 -- 172.21.15.120:0/3773961765 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb10138d90 con 0x7ffb10142680 2024-02-06T22:17:59.053 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.052+0000 7ffb05ffb640 1 -- 172.21.15.120:0/3773961765 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ffb08014820 con 0x7ffb10142680 2024-02-06T22:17:59.054 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.052+0000 7ffb05ffb640 1 --2- 172.21.15.120:0/3773961765 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ffaf406d1a0 0x7ffaf406f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:17:59.054 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.052+0000 7ffb150d9640 1 --2- 172.21.15.120:0/3773961765 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ffaf406d1a0 0x7ffaf406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:17:59.054 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.052+0000 7ffb150d9640 10 monclient: get_auth_request con 0x7ffaf406d1a0 auth_method 0 2024-02-06T22:17:59.054 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.053+0000 7ffb150d9640 1 --2- 172.21.15.120:0/3773961765 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ffaf406d1a0 0x7ffaf406f660 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ffb00001780 tx=0x7ffb00009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:17:59.054 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.053+0000 7ffb05ffb640 10 client.18689.objecter ms_handle_connect 0x7ffaf406d1a0 2024-02-06T22:17:59.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.055+0000 7ffb05ffb640 1 -- 172.21.15.120:0/3773961765 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7ffb08059c00 con 0x7ffb10142680 2024-02-06T22:17:59.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.055+0000 7ffb05ffb640 10 client.18689.objecter ms_dispatch 0x7ffb10000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:17:59.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.055+0000 7ffb05ffb640 3 client.18689.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:17:59.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.055+0000 7ffb05ffb640 3 client.18689.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:17:59.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.055+0000 7ffb05ffb640 20 client.18689.objecter dump_active .. 0 homeless 2024-02-06T22:17:59.296 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.294+0000 7ffb05ffb640 1 -- 172.21.15.120:0/3773961765 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ffb08055ea0 con 0x7ffb10142680 2024-02-06T22:17:59.296 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.294+0000 7ffb05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:17:59.296 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.294+0000 7ffb05ffb640 10 monclient: _finish_command 1 = system:0 2024-02-06T22:17:59.468 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.467+0000 7ffb1733d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-02-06T22:17:59.468 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.467+0000 7ffb1733d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T22:17:59.468 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.467+0000 7ffb1733d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:17:59.468 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.467+0000 7ffb1733d640 1 -- 172.21.15.120:0/3773961765 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb101387f0 con 0x7ffb10142680 2024-02-06T22:17:59.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.507+0000 7ffb05ffb640 1 -- 172.21.15.120:0/3773961765 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1346) v1 ==== 74+0+17873 (secure 0 0 0) 0x7ffb101387f0 con 0x7ffb10142680 2024-02-06T22:17:59.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.507+0000 7ffb05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-02-06T22:17:59.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.507+0000 7ffb05ffb640 10 monclient: _finish_command 2 = system:0 2024-02-06T22:17:59.509 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T22:17:59.509 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":1346,"fsid":"4e4791bf-0a02-4478-9517-2d2e893f4f7c","created":"2024-02-06T21:40:31.800075+0000","modified":"2024-02-06T22:17:58.721570+0000","last_up_change":"2024-02-06T21:41:29.190309+0000","last_in_change":"2024-02-06T21:40:37.354288+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":115,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":341,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-06T21:41:20.093970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-02-06T21:41:33.186378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-02-06T21:42:36.837814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4bd0f591-2cf1-42da-8ac1-dce3188580c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1341,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6817","nonce":2062960776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6819","nonce":2062960776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6823","nonce":2062960776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2062960776},{"type":"v1","addr":"172.21.15.120:6821","nonce":2062960776}]},"public_addr":"172.21.15.120:6817/2062960776","cluster_addr":"172.21.15.120:6819/2062960776","heartbeat_back_addr":"172.21.15.120:6823/2062960776","heartbeat_front_addr":"172.21.15.120:6821/2062960776","state":["exists","up"]},{"osd":1,"uuid":"81dc801c-3768-4c2c-9adb-6c8300a341f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1327,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6809","nonce":3348315891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6811","nonce":3348315891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6815","nonce":3348315891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3348315891},{"type":"v1","addr":"172.21.15.120:6813","nonce":3348315891}]},"public_addr":"172.21.15.120:6809/3348315891","cluster_addr":"172.21.15.120:6811/3348315891","heartbeat_back_addr":"172.21.15.120:6815/3348315891","heartbeat_front_addr":"172.21.15.120:6813/3348315891","state":["exists","up"]},{"osd":2,"uuid":"9289721b-c13f-4a17-be68-3698090f69af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":1334,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6825","nonce":3229495265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6827","nonce":3229495265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6831","nonce":3229495265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3229495265},{"type":"v1","addr":"172.21.15.120:6829","nonce":3229495265}]},"public_addr":"172.21.15.120:6825/3229495265","cluster_addr":"172.21.15.120:6827/3229495265","heartbeat_back_addr":"172.21.15.120:6831/3229495265","heartbeat_front_addr":"172.21.15.120:6829/3229495265","state":["exists","up"]},{"osd":3,"uuid":"53534db0-2d4d-44ed-90e8-d67ea878b51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":1341,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6801","nonce":3324566974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6803","nonce":3324566974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6807","nonce":3324566974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":3324566974},{"type":"v1","addr":"172.21.15.120:6805","nonce":3324566974}]},"public_addr":"172.21.15.120:6801/3324566974","cluster_addr":"172.21.15.120:6803/3324566974","heartbeat_back_addr":"172.21.15.120:6807/3324566974","heartbeat_front_addr":"172.21.15.120:6805/3324566974","state":["exists","up"]},{"osd":4,"uuid":"e4205ab5-787c-44e0-9b96-273b8c9cf1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":1341,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6803","nonce":2191547918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6805","nonce":2191547918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6809","nonce":2191547918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2191547918},{"type":"v1","addr":"172.21.15.135:6807","nonce":2191547918}]},"public_addr":"172.21.15.135:6803/2191547918","cluster_addr":"172.21.15.135:6805/2191547918","heartbeat_back_addr":"172.21.15.135:6809/2191547918","heartbeat_front_addr":"172.21.15.135:6807/2191547918","state":["exists","up"]},{"osd":5,"uuid":"43b600bc-3afe-44bd-bd3f-690106e7e0ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":1341,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6819","nonce":563193066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6821","nonce":563193066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6826","nonce":563193066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":563193066},{"type":"v1","addr":"172.21.15.135:6823","nonce":563193066}]},"public_addr":"172.21.15.135:6819/563193066","cluster_addr":"172.21.15.135:6821/563193066","heartbeat_back_addr":"172.21.15.135:6826/563193066","heartbeat_front_addr":"172.21.15.135:6823/563193066","state":["exists","up"]},{"osd":6,"uuid":"1d6b9f96-1f9d-44f2-83b0-93f5c342fafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":1341,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6811","nonce":1080459988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6813","nonce":1080459988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6817","nonce":1080459988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":1080459988},{"type":"v1","addr":"172.21.15.135:6815","nonce":1080459988}]},"public_addr":"172.21.15.135:6811/1080459988","cluster_addr":"172.21.15.135:6813/1080459988","heartbeat_back_addr":"172.21.15.135:6817/1080459988","heartbeat_front_addr":"172.21.15.135:6815/1080459988","state":["exists","up"]},{"osd":7,"uuid":"ecae00f5-8e29-47dd-90d5-e51503bc9dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1341,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6825","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6827","nonce":3740758337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6829","nonce":3740758337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6833","nonce":3740758337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3740758337},{"type":"v1","addr":"172.21.15.135:6831","nonce":3740758337}]},"public_addr":"172.21.15.135:6827/3740758337","cluster_addr":"172.21.15.135:6829/3740758337","heartbeat_back_addr":"172.21.15.135:6833/3740758337","heartbeat_front_addr":"172.21.15.135:6831/3740758337","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.344503+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:17.073640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.245112+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.202999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:15.825107+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:15.918704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:16.203923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-06T21:41:15.753628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":4,"length":3}]},{"pool":34,"snaps":[{"begin":1,"length":1}]},{"pool":35,"snaps":[{"begin":3,"length":2}]},{"pool":63,"snaps":[{"begin":1,"length":1}]},{"pool":81,"snaps":[{"begin":1,"length":1}]},{"pool":97,"snaps":[{"begin":7,"length":2}]},{"pool":99,"snaps":[{"begin":1,"length":2}]},{"pool":107,"snaps":[{"begin":3,"length":1}]},{"pool":109,"snaps":[{"begin":2,"length":2}]},{"pool":121,"snaps":[{"begin":2,"length":3}]},{"pool":129,"snaps":[{"begin":3,"length":2}]},{"pool":138,"snaps":[{"begin":2,"length":2}]},{"pool":144,"snaps":[{"begin":2,"length":1},{"begin":4,"length":2}]},{"pool":146,"snaps":[{"begin":2,"length":1}]},{"pool":153,"snaps":[{"begin":2,"length":3}]},{"pool":163,"snaps":[{"begin":4,"length":1},{"begin":6,"length":2}]},{"pool":171,"snaps":[{"begin":2,"length":1}]},{"pool":177,"snaps":[{"begin":3,"length":1}]},{"pool":203,"snaps":[{"begin":7,"length":1}]},{"pool":207,"snaps":[{"begin":8,"length":1}]},{"pool":209,"snaps":[{"begin":10,"length":1}]},{"pool":210,"snaps":[{"begin":11,"length":2}]},{"pool":213,"snaps":[{"begin":14,"length":1}]},{"pool":256,"snaps":[{"begin":2,"length":1}]},{"pool":257,"snaps":[{"begin":3,"length":1}]},{"pool":260,"snaps":[{"begin":4,"length":1}]},{"pool":264,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-06T22:17:59.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 10 librados: watch_flush enter 2024-02-06T22:17:59.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 10 librados: watch_flush exit 2024-02-06T22:17:59.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 20 client.18689.objecter shutdown clearing up homeless session... 2024-02-06T22:17:59.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 10 client.18689.objecter successfully canceled tick 2024-02-06T22:17:59.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 1 -- 172.21.15.120:0/3773961765 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ffaf406d1a0 msgr2=0x7ffaf406f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:59.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 1 --2- 172.21.15.120:0/3773961765 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ffaf406d1a0 0x7ffaf406f660 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ffb00001780 tx=0x7ffb00009040 comp rx=0 tx=0).stop 2024-02-06T22:17:59.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 10 monclient: shutdown 2024-02-06T22:17:59.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:17:59.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 1 -- 172.21.15.120:0/3773961765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb10142680 msgr2=0x7ffb10136ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:17:59.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 1 --2- 172.21.15.120:0/3773961765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb10142680 0x7ffb10136ba0 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800eba0 tx=0x7ffb0800c490 comp rx=0 tx=0).stop 2024-02-06T22:17:59.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 1 -- 172.21.15.120:0/3773961765 shutdown_connections 2024-02-06T22:17:59.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 1 --2- 172.21.15.120:0/3773961765 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7ffaf406d1a0 0x7ffaf406f660 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:59.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 1 --2- 172.21.15.120:0/3773961765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb10142680 0x7ffb10136ba0 unknown :-1 s=CLOSED pgs=3341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:59.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 1 --2- 172.21.15.120:0/3773961765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb1013ff40 0x7ffb101331f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:59.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 1 --2- 172.21.15.120:0/3773961765 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb100b54b0 0x7ffb101365b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:17:59.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 1 -- 172.21.15.120:0/3773961765 >> 172.21.15.120:0/3773961765 conn(0x7ffb100b8660 msgr2=0x7ffb1013f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:17:59.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 1 -- 172.21.15.120:0/3773961765 shutdown_connections 2024-02-06T22:17:59.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 1 -- 172.21.15.120:0/3773961765 wait complete. 2024-02-06T22:17:59.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:17:59.509+0000 7ffb1733d640 1 librados: shutdown 2024-02-06T22:18:00.527 INFO:tasks.ceph:Scrubbing osd.0 2024-02-06T22:18:00.527 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-02-06T22:18:00.634 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.633+0000 7fe543bcf640 10 monclient: get_monmap_and_config 2024-02-06T22:18:00.634 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.633+0000 7fe543bcf640 10 monclient: build_initial_monmap 2024-02-06T22:18:00.634 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.633+0000 7fe543bcf640 10 monclient: monmap: 2024-02-06T22:18:00.634 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:00.634 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:00.634 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:00.634054+0000 2024-02-06T22:18:00.634 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:00.634054+0000 2024-02-06T22:18:00.634 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:00.635 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:00.635 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:00.635 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:00.635 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:00.635 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:00.635 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:00.635 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:00.638 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.634+0000 7fe543bcf640 1 Processor -- start 2024-02-06T22:18:00.638 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.634+0000 7fe543bcf640 1 -- start start 2024-02-06T22:18:00.638 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.634+0000 7fe543bcf640 10 monclient: init 2024-02-06T22:18:00.638 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:00.638 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-02-06T22:18:00.638 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe53c0b5240 0x7fe53c142fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:00.638 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 10 monclient(hunting): picked mon.noname-c con 0x7fe53c0b5240 addr v2:172.21.15.135:3300/0 2024-02-06T22:18:00.638 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe53c0b5620 0x7fe53c0afa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:00.638 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 10 monclient(hunting): picked mon.noname-a con 0x7fe53c0b5620 addr v2:172.21.15.120:3300/0 2024-02-06T22:18:00.638 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe53c0aef10 0x7fe53c0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:00.638 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 10 monclient(hunting): picked mon.noname-b con 0x7fe53c0aef10 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:00.638 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:00.638 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe53c0b2240 con 0x7fe53c0b5620 2024-02-06T22:18:00.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:00.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe53c0b13a0 con 0x7fe53c0b5240 2024-02-06T22:18:00.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:00.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c142760 con 0x7fe53c0aef10 2024-02-06T22:18:00.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:00.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe543bcf640 10 monclient(hunting): authenticate will time out at 3510.971773s 2024-02-06T22:18:00.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe54216c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe53c0aef10 0x7fe53c0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:00.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.635+0000 7fe54116a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe53c0b5620 0x7fe53c0afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:00.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54196b640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe53c0b5240 0x7fe53c142fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:00.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54116a640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe53c0b5620 0x7fe53c0afa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:41544/0 (socket says 172.21.15.120:41544) 2024-02-06T22:18:00.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54116a640 1 -- 172.21.15.120:0/2850132484 learned_addr learned my addr 172.21.15.120:0/2850132484 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:00.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54116a640 10 monclient(hunting): get_auth_request con 0x7fe53c0b5620 auth_method 0 2024-02-06T22:18:00.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54116a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:00.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54116a640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:00.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54116a640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:00.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54216c640 10 monclient(hunting): get_auth_request con 0x7fe53c0aef10 auth_method 0 2024-02-06T22:18:00.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54216c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:00.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54216c640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:00.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54216c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:00.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54196b640 10 monclient(hunting): get_auth_request con 0x7fe53c0b5240 auth_method 0 2024-02-06T22:18:00.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54196b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:00.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54196b640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:00.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54196b640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:00.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54116a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:00.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54116a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:00.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54116a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:00.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54216c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:00.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54216c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:00.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54216c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:00.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54196b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:00.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54196b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:00.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.636+0000 7fe54196b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:00.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.638+0000 7fe54216c640 10 monclient(hunting): handle_auth_done global_id 18692 payload 274 2024-02-06T22:18:00.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.638+0000 7fe54216c640 1 -- 172.21.15.120:0/2850132484 >> v2:172.21.15.135:3300/0 conn(0x7fe53c0b5240 msgr2=0x7fe53c142fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:00.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.638+0000 7fe54216c640 1 --2- 172.21.15.120:0/2850132484 >> v2:172.21.15.135:3300/0 conn(0x7fe53c0b5240 0x7fe53c142fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:00.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.638+0000 7fe54216c640 1 -- 172.21.15.120:0/2850132484 >> v2:172.21.15.120:3300/0 conn(0x7fe53c0b5620 msgr2=0x7fe53c0afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:00.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.638+0000 7fe54216c640 1 --2- 172.21.15.120:0/2850132484 >> v2:172.21.15.120:3300/0 conn(0x7fe53c0b5620 0x7fe53c0afa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:00.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.638+0000 7fe54216c640 10 monclient: _finish_hunting 0 2024-02-06T22:18:00.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.638+0000 7fe54216c640 1 monclient: found mon.noname-b 2024-02-06T22:18:00.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.638+0000 7fe54216c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:00.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.638+0000 7fe54216c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T22:18:00.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.638+0000 7fe54216c640 1 -- 172.21.15.120:0/2850132484 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c0b15a0 con 0x7fe53c0aef10 2024-02-06T22:18:00.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.640+0000 7fe54216c640 1 --2- 172.21.15.120:0/2850132484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe53c0aef10 0x7fe53c0affa0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7fe53c06de20 tx=0x7fe53402bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af387fec0fcd7f24 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:00.644 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe540969640 1 -- 172.21.15.120:0/2850132484 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53402cb40 con 0x7fe53c0aef10 2024-02-06T22:18:00.644 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe540969640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:00.644 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe540969640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:18:00.644 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe540969640 10 monclient: dump: 2024-02-06T22:18:00.644 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:00.644 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:00.644 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.644 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.644 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:00.645 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:00.645 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:00.645 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:00.645 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:00.645 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:00.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe540969640 10 monclient: _finish_auth 0 2024-02-06T22:18:00.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe540969640 10 monclient: _check_auth_tickets 2024-02-06T22:18:00.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe540969640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:00.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe540969640 1 -- 172.21.15.120:0/2850132484 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53402cca0 con 0x7fe53c0aef10 2024-02-06T22:18:00.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe543bcf640 5 monclient: authenticate success, global_id 18692 2024-02-06T22:18:00.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe543bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T22:18:00.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe540969640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:00.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe543bcf640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:00.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe540969640 1 -- 172.21.15.120:0/2850132484 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe534034660 con 0x7fe53c0aef10 2024-02-06T22:18:00.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe540969640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:00.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe543bcf640 4 set_mon_vals no callback set 2024-02-06T22:18:00.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe540969640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T22:18:00.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.643+0000 7fe540969640 10 monclient: dump: 2024-02-06T22:18:00.646 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:00.646 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:00.646 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.646 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.646 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:00.646 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:00.646 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:00.646 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:00.646 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:00.646 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:00.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.644+0000 7fe543bcf640 10 monclient: shutdown 2024-02-06T22:18:00.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.644+0000 7fe543bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:00.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.644+0000 7fe543bcf640 1 -- 172.21.15.120:0/2850132484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe53c0aef10 msgr2=0x7fe53c0affa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:00.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.644+0000 7fe543bcf640 1 --2- 172.21.15.120:0/2850132484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe53c0aef10 0x7fe53c0affa0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7fe53c06de20 tx=0x7fe53402bd90 comp rx=0 tx=0).stop 2024-02-06T22:18:00.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.644+0000 7fe543bcf640 1 -- 172.21.15.120:0/2850132484 shutdown_connections 2024-02-06T22:18:00.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.644+0000 7fe543bcf640 1 --2- 172.21.15.120:0/2850132484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe53c0aef10 0x7fe53c0affa0 unknown :-1 s=CLOSED pgs=3342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:00.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.644+0000 7fe543bcf640 1 --2- 172.21.15.120:0/2850132484 >> v2:172.21.15.120:3300/0 conn(0x7fe53c0b5620 0x7fe53c0afa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:00.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.644+0000 7fe543bcf640 1 --2- 172.21.15.120:0/2850132484 >> v2:172.21.15.135:3300/0 conn(0x7fe53c0b5240 0x7fe53c142fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:00.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.644+0000 7fe543bcf640 1 -- 172.21.15.120:0/2850132484 >> 172.21.15.120:0/2850132484 conn(0x7fe53c0b8660 msgr2=0x7fe53c0b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:00.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.644+0000 7fe543bcf640 1 -- 172.21.15.120:0/2850132484 shutdown_connections 2024-02-06T22:18:00.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.644+0000 7fe543bcf640 1 -- 172.21.15.120:0/2850132484 wait complete. 2024-02-06T22:18:00.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.645+0000 7fe543bcf640 10 monclient: build_initial_monmap 2024-02-06T22:18:00.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.645+0000 7fe543bcf640 10 monclient: monmap: 2024-02-06T22:18:00.651 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:00.651 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:00.651 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:00.651 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:00.651 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:00.651 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:00.651 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:00.651 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:00.651 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:00.651 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:00.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.645+0000 7fe543bcf640 1 librados: starting msgr at 2024-02-06T22:18:00.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.645+0000 7fe543bcf640 1 librados: starting objecter 2024-02-06T22:18:00.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.645+0000 7fe543bcf640 1 Processor -- start 2024-02-06T22:18:00.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.645+0000 7fe543bcf640 1 -- start start 2024-02-06T22:18:00.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.645+0000 7fe543bcf640 1 librados: setting wanted keys 2024-02-06T22:18:00.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.645+0000 7fe543bcf640 1 librados: calling monclient init 2024-02-06T22:18:00.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.645+0000 7fe543bcf640 10 monclient: init 2024-02-06T22:18:00.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:00.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T22:18:00.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe53c0aef10 0x7fe53c145e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:00.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 10 monclient(hunting): picked mon.noname-b con 0x7fe53c0aef10 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:00.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe53c0b5240 0x7fe53c0ab790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:00.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 10 monclient(hunting): picked mon.noname-a con 0x7fe53c0b5240 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:00.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe53c0ab280 0x7fe53c13b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:00.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 10 monclient(hunting): picked mon.noname-c con 0x7fe53c0ab280 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:00.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:00.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c0ad150 con 0x7fe53c0b5240 2024-02-06T22:18:00.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:00.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c0abdb0 con 0x7fe53c0aef10 2024-02-06T22:18:00.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:00.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c13b6c0 con 0x7fe53c0ab280 2024-02-06T22:18:00.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:00.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe543bcf640 10 monclient(hunting): authenticate will time out at 3510.982818s 2024-02-06T22:18:00.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe54216c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe53c0ab280 0x7fe53c13b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:00.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe54116a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe53c0b5240 0x7fe53c0ab790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:00.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.646+0000 7fe54196b640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe53c0aef10 0x7fe53c145e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:00.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54116a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe53c0b5240 0x7fe53c0ab790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:41554/0 (socket says 172.21.15.120:41554) 2024-02-06T22:18:00.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54116a640 1 -- 172.21.15.120:0/1298227786 learned_addr learned my addr 172.21.15.120:0/1298227786 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:00.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54116a640 10 monclient(hunting): get_auth_request con 0x7fe53c0b5240 auth_method 0 2024-02-06T22:18:00.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54116a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:00.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54116a640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:00.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54116a640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:00.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54216c640 10 monclient(hunting): get_auth_request con 0x7fe53c0ab280 auth_method 0 2024-02-06T22:18:00.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54216c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:00.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54216c640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:00.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54216c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:00.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54196b640 10 monclient(hunting): get_auth_request con 0x7fe53c0aef10 auth_method 0 2024-02-06T22:18:00.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54196b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:00.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54196b640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:00.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54196b640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:00.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54116a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:00.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54116a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:00.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54116a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:00.655 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54216c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:00.655 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54216c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:00.655 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54216c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:00.655 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54196b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:00.655 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54196b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:00.655 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.647+0000 7fe54196b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:00.657 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.650+0000 7fe54216c640 10 monclient(hunting): handle_auth_done global_id 18695 payload 824 2024-02-06T22:18:00.657 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.650+0000 7fe54216c640 1 -- 172.21.15.120:0/1298227786 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe53c0aef10 msgr2=0x7fe53c145e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:00.657 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.650+0000 7fe54216c640 1 --2- 172.21.15.120:0/1298227786 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe53c0aef10 0x7fe53c145e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:00.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.650+0000 7fe54216c640 1 -- 172.21.15.120:0/1298227786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe53c0b5240 msgr2=0x7fe53c0ab790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:00.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.650+0000 7fe54216c640 1 --2- 172.21.15.120:0/1298227786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe53c0b5240 0x7fe53c0ab790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:00.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.650+0000 7fe54216c640 10 monclient: _finish_hunting 0 2024-02-06T22:18:00.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.650+0000 7fe54216c640 1 monclient: found mon.noname-c 2024-02-06T22:18:00.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.650+0000 7fe54216c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:00.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.650+0000 7fe54216c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T22:18:00.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.650+0000 7fe54216c640 1 -- 172.21.15.120:0/1298227786 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe534009a70 con 0x7fe53c0ab280 2024-02-06T22:18:00.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.650+0000 7fe54116a640 1 --2- 172.21.15.120:0/1298227786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe53c0b5240 0x7fe53c0ab790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-06T22:18:00.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.652+0000 7fe54216c640 1 --2- 172.21.15.120:0/1298227786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe53c0ab280 0x7fe53c13b180 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7fe534033000 tx=0x7fe534004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:00.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.652+0000 7fe5327fc640 10 client.?.objecter ms_handle_connect 0x7fe53c0ab280 2024-02-06T22:18:00.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.652+0000 7fe5327fc640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:00.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe5327fc640 1 -- 172.21.15.120:0/1298227786 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53403e610 con 0x7fe53c0ab280 2024-02-06T22:18:00.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe5327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:00.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe5327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T22:18:00.659 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe5327fc640 10 monclient: dump: 2024-02-06T22:18:00.659 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:00.659 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:00.659 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.659 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.659 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:00.659 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:00.659 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:00.659 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:00.659 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:00.659 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:00.659 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe5327fc640 10 monclient: _finish_auth 0 2024-02-06T22:18:00.659 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe5327fc640 10 monclient: _check_auth_tickets 2024-02-06T22:18:00.659 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe5327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:00.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe5327fc640 1 -- 172.21.15.120:0/1298227786 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53403e770 con 0x7fe53c0ab280 2024-02-06T22:18:00.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe543bcf640 5 monclient: authenticate success, global_id 18695 2024-02-06T22:18:00.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe543bcf640 10 monclient: _renew_subs 2024-02-06T22:18:00.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe543bcf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:18:00.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe543bcf640 1 -- 172.21.15.120:0/1298227786 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53c13b8e0 con 0x7fe53c0ab280 2024-02-06T22:18:00.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe5327fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:00.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe5327fc640 1 -- 172.21.15.120:0/1298227786 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53403eb20 con 0x7fe53c0ab280 2024-02-06T22:18:00.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe5327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:00.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe5327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T22:18:00.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe5327fc640 10 monclient: dump: 2024-02-06T22:18:00.660 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:00.660 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:00.660 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.660 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.661 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:00.661 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:00.661 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:00.661 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:00.661 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:00.661 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:00.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe533fff640 4 set_mon_vals no callback set 2024-02-06T22:18:00.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe543bcf640 10 client.18695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:00.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe543bcf640 10 monclient: _renew_subs 2024-02-06T22:18:00.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe543bcf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:18:00.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe543bcf640 1 -- 172.21.15.120:0/1298227786 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53c1dae90 con 0x7fe53c0ab280 2024-02-06T22:18:00.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.656+0000 7fe543bcf640 1 librados: init done 2024-02-06T22:18:00.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.657+0000 7fe543bcf640 10 client.18695.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:00.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.657+0000 7fe543bcf640 15 client.18695.objecter _session_command_op_assign -1 1 2024-02-06T22:18:00.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.657+0000 7fe543bcf640 20 client.18695.objecter _get_session osd=-1 returning homeless 2024-02-06T22:18:00.662 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.657+0000 7fe543bcf640 10 client.18695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:00.662 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.657+0000 7fe543bcf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:18:00.662 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.657+0000 7fe543bcf640 1 -- 172.21.15.120:0/1298227786 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe53c0b04e0 con 0x7fe53c0ab280 2024-02-06T22:18:00.679 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.678+0000 7fe5327fc640 1 -- 172.21.15.120:0/1298227786 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe53403b050 con 0x7fe53c0ab280 2024-02-06T22:18:00.679 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.678+0000 7fe5327fc640 1 --2- 172.21.15.120:0/1298227786 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe52006d1f0 0x7fe52006f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:00.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.679+0000 7fe54196b640 1 --2- 172.21.15.120:0/1298227786 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe52006d1f0 0x7fe52006f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:00.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.679+0000 7fe54196b640 10 monclient: get_auth_request con 0x7fe52006d1f0 auth_method 0 2024-02-06T22:18:00.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.679+0000 7fe5327fc640 10 client.18695.objecter ms_handle_connect 0x7fe52006d1f0 2024-02-06T22:18:00.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.679+0000 7fe54196b640 1 --2- 172.21.15.120:0/1298227786 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe52006d1f0 0x7fe52006f6b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe538009ac0 tx=0x7fe538009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:00.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 1 -- 172.21.15.120:0/1298227786 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fe53407b6c0 con 0x7fe53c0ab280 2024-02-06T22:18:00.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 10 client.18695.objecter ms_dispatch 0x7fe53c000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:00.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 3 client.18695.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:00.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 3 client.18695.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:00.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 10 client.18695.objecter checking command 1 2024-02-06T22:18:00.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 1 --2- 172.21.15.120:0/1298227786 >> [v2:172.21.15.120:6816/2062960776,v1:172.21.15.120:6817/2062960776] conn(0x7fe520074f80 0x7fe520077420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:00.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 20 client.18695.objecter _get_session s=0x7fe5200748b0 osd=0 3 2024-02-06T22:18:00.687 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 20 client.18695.objecter put_session s=0x7fe5200748b0 osd=0 3 2024-02-06T22:18:00.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 15 client.18695.objecter _session_command_op_remove -1 1 2024-02-06T22:18:00.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 20 client.18695.objecter _get_session s=0x7fe5200748b0 osd=0 3 2024-02-06T22:18:00.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 20 client.18695.objecter get_session s=0x7fe5200748b0 osd=0 3 2024-02-06T22:18:00.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 15 client.18695.objecter _session_command_op_assign 0 1 2024-02-06T22:18:00.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 20 client.18695.objecter put_session s=0x7fe5200748b0 osd=0 4 2024-02-06T22:18:00.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 10 client.18695.objecter _send_command 1 2024-02-06T22:18:00.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 1 -- 172.21.15.120:0/1298227786 --> [v2:172.21.15.120:6816/2062960776,v1:172.21.15.120:6817/2062960776] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe520077af0 con 0x7fe520074f80 2024-02-06T22:18:00.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 20 client.18695.objecter dump_active .. 0 homeless 2024-02-06T22:18:00.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 1 -- 172.21.15.120:0/1298227786 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_get_version_reply(handle=1 version=1346) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe534077960 con 0x7fe53c0ab280 2024-02-06T22:18:00.688 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.686+0000 7fe5327fc640 10 monclient: handle_get_version_reply finishing 1 version 1346 2024-02-06T22:18:00.705 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.705+0000 7fe54116a640 1 --2- 172.21.15.120:0/1298227786 >> [v2:172.21.15.120:6816/2062960776,v1:172.21.15.120:6817/2062960776] conn(0x7fe520074f80 0x7fe520077420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:00.706 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.706+0000 7fe54116a640 10 monclient: get_auth_request con 0x7fe520074f80 auth_method 0 2024-02-06T22:18:00.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.730+0000 7fe5327fc640 10 client.18695.objecter ms_handle_connect 0x7fe520074f80 2024-02-06T22:18:00.731 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.730+0000 7fe54116a640 1 --2- 172.21.15.120:0/1298227786 >> [v2:172.21.15.120:6816/2062960776,v1:172.21.15.120:6817/2062960776] conn(0x7fe520074f80 0x7fe520077420 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:00.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.759+0000 7fe5327fc640 1 -- 172.21.15.120:0/1298227786 <== osd.0 v2:172.21.15.120:6816/2062960776 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe520077af0 con 0x7fe520074f80 2024-02-06T22:18:00.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.759+0000 7fe5327fc640 10 client.18695.objecter ms_dispatch 0x7fe53c000b90 command_reply(tid 1: 0 ) v1 2024-02-06T22:18:00.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.759+0000 7fe5327fc640 10 client.18695.objecter _finish_command 1 = system:0 2024-02-06T22:18:00.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.759+0000 7fe5327fc640 20 client.18695.objecter put_session s=0x7fe5200748b0 osd=0 4 2024-02-06T22:18:00.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.759+0000 7fe5327fc640 15 client.18695.objecter _session_command_op_remove 0 1 2024-02-06T22:18:00.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.774+0000 7fe543bcf640 10 client.18695.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-02-06T22:18:00.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.774+0000 7fe543bcf640 15 client.18695.objecter _session_command_op_assign -1 2 2024-02-06T22:18:00.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.774+0000 7fe543bcf640 20 client.18695.objecter _get_session s=0x7fe5200748b0 osd=0 3 2024-02-06T22:18:00.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.774+0000 7fe543bcf640 20 client.18695.objecter put_session s=0x7fe5200748b0 osd=0 3 2024-02-06T22:18:00.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.774+0000 7fe543bcf640 20 client.18695.objecter _get_session s=0x7fe5200748b0 osd=0 3 2024-02-06T22:18:00.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.774+0000 7fe543bcf640 15 client.18695.objecter _session_command_op_remove -1 2 2024-02-06T22:18:00.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.774+0000 7fe543bcf640 20 client.18695.objecter get_session s=0x7fe5200748b0 osd=0 3 2024-02-06T22:18:00.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.774+0000 7fe543bcf640 15 client.18695.objecter _session_command_op_assign 0 2 2024-02-06T22:18:00.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.774+0000 7fe543bcf640 20 client.18695.objecter put_session s=0x7fe5200748b0 osd=0 4 2024-02-06T22:18:00.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.774+0000 7fe543bcf640 10 client.18695.objecter _send_command 2 2024-02-06T22:18:00.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.774+0000 7fe543bcf640 1 -- 172.21.15.120:0/1298227786 --> [v2:172.21.15.120:6816/2062960776,v1:172.21.15.120:6817/2062960776] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe53c0b2430 con 0x7fe520074f80 2024-02-06T22:18:00.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe5327fc640 1 -- 172.21.15.120:0/1298227786 <== osd.0 v2:172.21.15.120:6816/2062960776 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe52c016420 con 0x7fe520074f80 2024-02-06T22:18:00.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe5327fc640 10 client.18695.objecter ms_dispatch 0x7fe53c000b90 command_reply(tid 2: 0 ) v1 2024-02-06T22:18:00.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe5327fc640 10 client.18695.objecter _finish_command 2 = system:0 2024-02-06T22:18:00.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe5327fc640 20 client.18695.objecter put_session s=0x7fe5200748b0 osd=0 4 2024-02-06T22:18:00.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe5327fc640 15 client.18695.objecter _session_command_op_remove 0 2 2024-02-06T22:18:00.789 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-02-06T22:18:00.789 INFO:teuthology.orchestra.run.smithi120.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-06T22:18:00.789 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-02-06T22:18:00.791 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 10 librados: watch_flush enter 2024-02-06T22:18:00.791 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 10 librados: watch_flush exit 2024-02-06T22:18:00.791 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 10 client.18695.objecter close_session for osd.0 2024-02-06T22:18:00.791 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 1 -- 172.21.15.120:0/1298227786 >> [v2:172.21.15.120:6816/2062960776,v1:172.21.15.120:6817/2062960776] conn(0x7fe520074f80 msgr2=0x7fe520077420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:00.791 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 1 --2- 172.21.15.120:0/1298227786 >> [v2:172.21.15.120:6816/2062960776,v1:172.21.15.120:6817/2062960776] conn(0x7fe520074f80 0x7fe520077420 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:00.791 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 20 client.18695.objecter put_session s=0x7fe5200748b0 osd=0 1 2024-02-06T22:18:00.791 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 20 client.18695.objecter shutdown clearing up homeless session... 2024-02-06T22:18:00.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 10 client.18695.objecter successfully canceled tick 2024-02-06T22:18:00.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 1 -- 172.21.15.120:0/1298227786 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe52006d1f0 msgr2=0x7fe52006f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:00.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 1 --2- 172.21.15.120:0/1298227786 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fe52006d1f0 0x7fe52006f6b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe538009ac0 tx=0x7fe538009190 comp rx=0 tx=0).stop 2024-02-06T22:18:00.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 10 monclient: shutdown 2024-02-06T22:18:00.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:00.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 1 -- 172.21.15.120:0/1298227786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe53c0ab280 msgr2=0x7fe53c13b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:00.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 1 --2- 172.21.15.120:0/1298227786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe53c0ab280 0x7fe53c13b180 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7fe534033000 tx=0x7fe534004a40 comp rx=0 tx=0).stop 2024-02-06T22:18:00.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe54216c640 1 -- 172.21.15.120:0/1298227786 reap_dead start 2024-02-06T22:18:00.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 1 -- 172.21.15.120:0/1298227786 shutdown_connections 2024-02-06T22:18:00.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 1 -- 172.21.15.120:0/1298227786 >> 172.21.15.120:0/1298227786 conn(0x7fe53c0b8660 msgr2=0x7fe53c1414f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:00.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 1 -- 172.21.15.120:0/1298227786 shutdown_connections 2024-02-06T22:18:00.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 1 -- 172.21.15.120:0/1298227786 wait complete. 2024-02-06T22:18:00.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.788+0000 7fe543bcf640 1 librados: shutdown 2024-02-06T22:18:00.799 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-02-06T22:18:00.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.907+0000 7f516a9cd640 10 monclient: get_monmap_and_config 2024-02-06T22:18:00.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.907+0000 7f516a9cd640 10 monclient: build_initial_monmap 2024-02-06T22:18:00.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.908+0000 7f516a9cd640 10 monclient: monmap: 2024-02-06T22:18:00.908 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:00.908 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:00.909 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:00.908271+0000 2024-02-06T22:18:00.909 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:00.908271+0000 2024-02-06T22:18:00.909 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:00.909 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:00.909 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:00.909 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:00.909 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:00.909 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:00.909 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:00.909 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:00.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.908+0000 7f516a9cd640 1 Processor -- start 2024-02-06T22:18:00.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.908+0000 7f516a9cd640 1 -- start start 2024-02-06T22:18:00.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.908+0000 7f516a9cd640 10 monclient: init 2024-02-06T22:18:00.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.909+0000 7f516a9cd640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:00.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.909+0000 7f516a9cd640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-02-06T22:18:00.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.909+0000 7f516a9cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f51640b54b0 addr v1:172.21.15.135:6789/0 2024-02-06T22:18:00.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.909+0000 7f516a9cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f516413ff40 addr v1:172.21.15.120:6789/0 2024-02-06T22:18:00.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.909+0000 7f516a9cd640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5164142d90 0x7f51640a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:00.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.909+0000 7f516a9cd640 10 monclient(hunting): picked mon.noname-c con 0x7f5164142d90 addr v2:172.21.15.135:3300/0 2024-02-06T22:18:00.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.909+0000 7f516a9cd640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5164143200 con 0x7f516413ff40 2024-02-06T22:18:00.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.909+0000 7f516a9cd640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51640ab720 con 0x7f51640b54b0 2024-02-06T22:18:00.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.909+0000 7f516a9cd640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:00.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.909+0000 7f516a9cd640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51640ab520 con 0x7f5164142d90 2024-02-06T22:18:00.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.909+0000 7f516a9cd640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:00.911 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.909+0000 7f516a9cd640 10 monclient(hunting): authenticate will time out at 3511.245937s 2024-02-06T22:18:00.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.945+0000 7f5168f6a640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f5164142d90 0x7f51640a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:00.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.945+0000 7f5163fff640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f51640b54b0 0x7f516413f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.120:37152/0 (socket says 172.21.15.120:37152) 2024-02-06T22:18:00.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.945+0000 7f5163fff640 1 -- 172.21.15.120:0/1444282263 learned_addr learned my addr 172.21.15.120:0/1444282263 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T22:18:00.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.945+0000 7f5168f6a640 10 monclient(hunting): get_auth_request con 0x7f5164142d90 auth_method 0 2024-02-06T22:18:00.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.945+0000 7f5168f6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:00.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.945+0000 7f5168f6a640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:00.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.945+0000 7f5168f6a640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:00.947 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.946+0000 7f5168f6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:00.947 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.946+0000 7f5168f6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:00.947 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.946+0000 7f5168f6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:00.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.948+0000 7f5168f6a640 10 monclient(hunting): handle_auth_done global_id 21031 payload 274 2024-02-06T22:18:00.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.948+0000 7f5168f6a640 1 -- 172.21.15.120:0/1444282263 >> v1:172.21.15.135:6789/0 conn(0x7f51640b54b0 legacy=0x7f516413f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:00.948 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.948+0000 7f5168f6a640 1 -- 172.21.15.120:0/1444282263 >> v1:172.21.15.120:6789/0 conn(0x7f516413ff40 legacy=0x7f5164142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:00.949 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.948+0000 7f5168f6a640 10 monclient: _finish_hunting 0 2024-02-06T22:18:00.949 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.948+0000 7f5168f6a640 1 monclient: found mon.noname-c 2024-02-06T22:18:00.949 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.948+0000 7f5168f6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:00.949 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.948+0000 7f5168f6a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T22:18:00.949 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.948+0000 7f5168f6a640 1 -- 172.21.15.120:0/1444282263 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51640b4800 con 0x7f5164142d90 2024-02-06T22:18:00.953 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.953+0000 7f5168f6a640 1 --2- 172.21.15.120:0/1444282263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5164142d90 0x7f51640a9860 secure :-1 s=READY pgs=5088 cs=0 l=1 rev1=1 crypto rx=0x7f515c009d90 tx=0x7f515c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2373186024bed369 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:00.969 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f5162ffd640 1 -- 172.21.15.120:0/1444282263 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c02cc60 con 0x7f5164142d90 2024-02-06T22:18:00.969 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f5162ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:00.969 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f5162ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T22:18:00.969 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f5162ffd640 10 monclient: dump: 2024-02-06T22:18:00.970 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:00.970 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:00.970 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.970 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.970 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:00.970 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:00.970 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:00.970 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:00.970 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:00.970 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:00.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f5162ffd640 10 monclient: _finish_auth 0 2024-02-06T22:18:00.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f5162ffd640 10 monclient: _check_auth_tickets 2024-02-06T22:18:00.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f5162ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:00.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f5162ffd640 1 -- 172.21.15.120:0/1444282263 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c02cdc0 con 0x7f5164142d90 2024-02-06T22:18:00.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f5162ffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:00.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f5162ffd640 1 -- 172.21.15.120:0/1444282263 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c034660 con 0x7f5164142d90 2024-02-06T22:18:00.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f516a9cd640 5 monclient: authenticate success, global_id 21031 2024-02-06T22:18:00.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f516a9cd640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:00.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f516a9cd640 4 set_mon_vals no callback set 2024-02-06T22:18:00.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f5162ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:00.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f5162ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T22:18:00.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f5162ffd640 10 monclient: dump: 2024-02-06T22:18:00.971 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:00.971 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:00.971 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.971 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.971 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:00.972 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:00.972 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:00.972 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:00.972 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:00.972 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:00.974 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f516a9cd640 10 monclient: shutdown 2024-02-06T22:18:00.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f516a9cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:00.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f516a9cd640 1 -- 172.21.15.120:0/1444282263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5164142d90 msgr2=0x7f51640a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:00.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.969+0000 7f516a9cd640 1 --2- 172.21.15.120:0/1444282263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5164142d90 0x7f51640a9860 secure :-1 s=READY pgs=5088 cs=0 l=1 rev1=1 crypto rx=0x7f515c009d90 tx=0x7f515c02beb0 comp rx=0 tx=0).stop 2024-02-06T22:18:00.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.970+0000 7f516a9cd640 1 -- 172.21.15.120:0/1444282263 shutdown_connections 2024-02-06T22:18:00.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.970+0000 7f516a9cd640 1 --2- 172.21.15.120:0/1444282263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5164142d90 0x7f51640a9860 unknown :-1 s=CLOSED pgs=5088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:00.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.970+0000 7f516a9cd640 1 -- 172.21.15.120:0/1444282263 >> 172.21.15.120:0/1444282263 conn(0x7f51640b8660 msgr2=0x7f51640b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:00.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.970+0000 7f516a9cd640 1 -- 172.21.15.120:0/1444282263 shutdown_connections 2024-02-06T22:18:00.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.970+0000 7f516a9cd640 1 -- 172.21.15.120:0/1444282263 wait complete. 2024-02-06T22:18:00.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.970+0000 7f516a9cd640 10 monclient: build_initial_monmap 2024-02-06T22:18:00.975 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.970+0000 7f516a9cd640 10 monclient: monmap: 2024-02-06T22:18:00.975 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:00.975 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:00.975 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:00.976 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:00.976 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:00.976 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:00.976 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:00.976 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:00.976 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:00.976 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:00.976 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.970+0000 7f516a9cd640 1 librados: starting msgr at 2024-02-06T22:18:00.976 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.970+0000 7f516a9cd640 1 librados: starting objecter 2024-02-06T22:18:00.976 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 1 Processor -- start 2024-02-06T22:18:00.976 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 1 -- start start 2024-02-06T22:18:00.976 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 1 librados: setting wanted keys 2024-02-06T22:18:00.976 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 1 librados: calling monclient init 2024-02-06T22:18:00.976 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 10 monclient: init 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51640b54b0 0x7f5164138ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 10 monclient(hunting): picked mon.noname-c con 0x7f51640b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f516413ff40 0x7f516414d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 10 monclient(hunting): picked mon.noname-b con 0x7f516413ff40 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5164142d90 0x7f51640ad350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 10 monclient(hunting): picked mon.noname-a con 0x7f5164142d90 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51640ac6b0 con 0x7f5164142d90 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51641368c0 con 0x7f516413ff40 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51640ac890 con 0x7f51640b54b0 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:00.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.971+0000 7f516a9cd640 10 monclient(hunting): authenticate will time out at 3511.308296s 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.972+0000 7f5163fff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51640b54b0 0x7f5164138ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.972+0000 7f5168f6a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5164142d90 0x7f51640ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.972+0000 7f5168f6a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5164142d90 0x7f51640ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:41568/0 (socket says 172.21.15.120:41568) 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.972+0000 7f5168f6a640 1 -- 172.21.15.120:0/2655087693 learned_addr learned my addr 172.21.15.120:0/2655087693 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.972+0000 7f5168f6a640 10 monclient(hunting): get_auth_request con 0x7f5164142d90 auth_method 0 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.972+0000 7f5168f6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.972+0000 7f5168f6a640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.972+0000 7f5168f6a640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.972+0000 7f5163fff640 10 monclient(hunting): get_auth_request con 0x7f51640b54b0 auth_method 0 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.972+0000 7f5163fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.972+0000 7f5163fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.972+0000 7f5163fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.973+0000 7f5168f6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.973+0000 7f5168f6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.973+0000 7f5168f6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:00.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.973+0000 7f5163fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:00.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.973+0000 7f5163fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:00.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.973+0000 7f5163fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:00.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.976+0000 7f5163fff640 10 monclient(hunting): handle_auth_done global_id 18698 payload 824 2024-02-06T22:18:00.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.976+0000 7f5163fff640 1 -- 172.21.15.120:0/2655087693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f516413ff40 msgr2=0x7f516414d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:00.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.976+0000 7f5163fff640 1 --2- 172.21.15.120:0/2655087693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f516413ff40 0x7f516414d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:00.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.976+0000 7f5163fff640 1 -- 172.21.15.120:0/2655087693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5164142d90 msgr2=0x7f51640ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:00.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.976+0000 7f5163fff640 1 --2- 172.21.15.120:0/2655087693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5164142d90 0x7f51640ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:00.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.976+0000 7f5163fff640 10 monclient: _finish_hunting 0 2024-02-06T22:18:00.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.976+0000 7f5163fff640 1 monclient: found mon.noname-c 2024-02-06T22:18:00.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.976+0000 7f5163fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:00.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.976+0000 7f5163fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T22:18:00.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.976+0000 7f5163fff640 1 -- 172.21.15.120:0/2655087693 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c009a70 con 0x7f51640b54b0 2024-02-06T22:18:00.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.978+0000 7f5163fff640 1 --2- 172.21.15.120:0/2655087693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51640b54b0 0x7f5164138ad0 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 crypto rx=0x7f515400c390 tx=0x7f515400c790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:00.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.978+0000 7f5160ff9640 10 client.?.objecter ms_handle_connect 0x7f51640b54b0 2024-02-06T22:18:00.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.978+0000 7f5160ff9640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:00.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f5160ff9640 1 -- 172.21.15.120:0/2655087693 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5154005ce0 con 0x7f51640b54b0 2024-02-06T22:18:00.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f5160ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:00.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f5160ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T22:18:00.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f5160ff9640 10 monclient: dump: 2024-02-06T22:18:00.983 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:00.984 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:00.984 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.984 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.984 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:00.984 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:00.984 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:00.984 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:00.984 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:00.984 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:00.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f5160ff9640 10 monclient: _finish_auth 0 2024-02-06T22:18:00.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f5160ff9640 10 monclient: _check_auth_tickets 2024-02-06T22:18:00.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f5160ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:00.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f5160ff9640 1 -- 172.21.15.120:0/2655087693 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5154005e40 con 0x7f51640b54b0 2024-02-06T22:18:00.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f516a9cd640 5 monclient: authenticate success, global_id 18698 2024-02-06T22:18:00.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f516a9cd640 10 monclient: _renew_subs 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f516a9cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f516a9cd640 1 -- 172.21.15.120:0/2655087693 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51640acb70 con 0x7f51640b54b0 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f5160ff9640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f5160ff9640 1 -- 172.21.15.120:0/2655087693 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5154007310 con 0x7f51640b54b0 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f5160ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f5160ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f5160ff9640 10 monclient: dump: 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:00.985 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:00.986 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:00.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f51627fc640 4 set_mon_vals no callback set 2024-02-06T22:18:00.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f516a9cd640 10 client.18698.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:00.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f516a9cd640 10 monclient: _renew_subs 2024-02-06T22:18:00.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f516a9cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:18:00.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f516a9cd640 1 -- 172.21.15.120:0/2655087693 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51640acf00 con 0x7f51640b54b0 2024-02-06T22:18:00.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.982+0000 7f516a9cd640 1 librados: init done 2024-02-06T22:18:00.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.983+0000 7f516a9cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:00.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.983+0000 7f516a9cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:00.987 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.983+0000 7f516a9cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:18:00.987 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:00.983+0000 7f516a9cd640 1 -- 172.21.15.120:0/2655087693 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516404ef90 con 0x7f51640b54b0 2024-02-06T22:18:01.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.007+0000 7f5160ff9640 1 -- 172.21.15.120:0/2655087693 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5154007470 con 0x7f51640b54b0 2024-02-06T22:18:01.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.007+0000 7f5160ff9640 1 --2- 172.21.15.120:0/2655087693 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f515006d220 0x7f515006f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:01.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.007+0000 7f51637fe640 1 --2- 172.21.15.120:0/2655087693 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f515006d220 0x7f515006f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:01.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.007+0000 7f51637fe640 10 monclient: get_auth_request con 0x7f515006d220 auth_method 0 2024-02-06T22:18:01.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.008+0000 7f51637fe640 1 --2- 172.21.15.120:0/2655087693 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f515006d220 0x7f515006f6e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5164136f50 tx=0x7f5158007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:01.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.008+0000 7f5160ff9640 10 client.18698.objecter ms_handle_connect 0x7f515006d220 2024-02-06T22:18:01.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.010+0000 7f5160ff9640 1 -- 172.21.15.120:0/2655087693 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f5154001000 con 0x7f51640b54b0 2024-02-06T22:18:01.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.010+0000 7f5160ff9640 10 client.18698.objecter ms_dispatch 0x7f5164000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:01.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.010+0000 7f5160ff9640 3 client.18698.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:01.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.010+0000 7f5160ff9640 3 client.18698.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:01.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.010+0000 7f5160ff9640 20 client.18698.objecter dump_active .. 0 homeless 2024-02-06T22:18:01.243 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.242+0000 7f5160ff9640 1 -- 172.21.15.120:0/2655087693 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5154054570 con 0x7f51640b54b0 2024-02-06T22:18:01.243 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.242+0000 7f5160ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:01.243 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.242+0000 7f5160ff9640 10 monclient: _finish_command 1 = system:0 2024-02-06T22:18:01.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.416+0000 7f516a9cd640 1 -- 172.21.15.120:0/2655087693 --> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f51640aa450 con 0x7f515006d220 2024-02-06T22:18:01.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.417+0000 7f5160ff9640 1 -- 172.21.15.120:0/2655087693 <== mgr.4100 v2:172.21.15.135:6800/315664144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f51580050d0 con 0x7f515006d220 2024-02-06T22:18:01.417 INFO:teuthology.orchestra.run.smithi120.stderr:instructed osd(s) 0 to deep-scrub 2024-02-06T22:18:01.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 10 librados: watch_flush enter 2024-02-06T22:18:01.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 10 librados: watch_flush exit 2024-02-06T22:18:01.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 20 client.18698.objecter shutdown clearing up homeless session... 2024-02-06T22:18:01.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 10 client.18698.objecter successfully canceled tick 2024-02-06T22:18:01.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 1 -- 172.21.15.120:0/2655087693 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f515006d220 msgr2=0x7f515006f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:01.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 1 --2- 172.21.15.120:0/2655087693 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f515006d220 0x7f515006f6e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5164136f50 tx=0x7f5158007990 comp rx=0 tx=0).stop 2024-02-06T22:18:01.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 10 monclient: shutdown 2024-02-06T22:18:01.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:01.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 1 -- 172.21.15.120:0/2655087693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51640b54b0 msgr2=0x7f5164138ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:01.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 1 --2- 172.21.15.120:0/2655087693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51640b54b0 0x7f5164138ad0 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 crypto rx=0x7f515400c390 tx=0x7f515400c790 comp rx=0 tx=0).stop 2024-02-06T22:18:01.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 1 -- 172.21.15.120:0/2655087693 shutdown_connections 2024-02-06T22:18:01.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 1 --2- 172.21.15.120:0/2655087693 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f515006d220 0x7f515006f6e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 1 --2- 172.21.15.120:0/2655087693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f516413ff40 0x7f516414d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 1 --2- 172.21.15.120:0/2655087693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5164142d90 0x7f51640ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 1 --2- 172.21.15.120:0/2655087693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51640b54b0 0x7f5164138ad0 unknown :-1 s=CLOSED pgs=3344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 1 -- 172.21.15.120:0/2655087693 >> 172.21.15.120:0/2655087693 conn(0x7f51640b8660 msgr2=0x7f516413f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:01.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 1 -- 172.21.15.120:0/2655087693 shutdown_connections 2024-02-06T22:18:01.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 1 -- 172.21.15.120:0/2655087693 wait complete. 2024-02-06T22:18:01.422 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.419+0000 7f516a9cd640 1 librados: shutdown 2024-02-06T22:18:01.433 INFO:tasks.ceph:Scrubbing osd.1 2024-02-06T22:18:01.434 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-02-06T22:18:01.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.537+0000 7fd408932640 10 monclient: get_monmap_and_config 2024-02-06T22:18:01.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.537+0000 7fd408932640 10 monclient: build_initial_monmap 2024-02-06T22:18:01.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.537+0000 7fd408932640 10 monclient: monmap: 2024-02-06T22:18:01.538 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:01.538 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:01.538 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:01.537458+0000 2024-02-06T22:18:01.538 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:01.537458+0000 2024-02-06T22:18:01.538 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:01.538 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:01.538 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:01.538 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:01.538 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:01.538 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:01.538 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:01.538 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:01.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.537+0000 7fd408932640 1 Processor -- start 2024-02-06T22:18:01.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.537+0000 7fd408932640 1 -- start start 2024-02-06T22:18:01.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.537+0000 7fd408932640 10 monclient: init 2024-02-06T22:18:01.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.538+0000 7fd408932640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:01.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.538+0000 7fd408932640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-02-06T22:18:01.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.538+0000 7fd408932640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd4040af4b0 0x7fd40413b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:01.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.538+0000 7fd408932640 10 monclient(hunting): picked mon.noname-a con 0x7fd4040af4b0 addr v2:172.21.15.120:3300/0 2024-02-06T22:18:01.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.538+0000 7fd408932640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd404071eb0 addr v1:172.21.15.120:6789/0 2024-02-06T22:18:01.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.538+0000 7fd408932640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd404142030 addr v1:172.21.15.135:6789/0 2024-02-06T22:18:01.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.538+0000 7fd408932640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd404070ef0 con 0x7fd404071eb0 2024-02-06T22:18:01.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.538+0000 7fd408932640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4040711d0 con 0x7fd404142030 2024-02-06T22:18:01.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.538+0000 7fd408932640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:01.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.538+0000 7fd408932640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4040af890 con 0x7fd4040af4b0 2024-02-06T22:18:01.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.538+0000 7fd408932640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:01.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.538+0000 7fd408932640 10 monclient(hunting): authenticate will time out at 3511.875199s 2024-02-06T22:18:01.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.539+0000 7fd401d9b640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fd404071eb0 0x7fd40413be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:50060/0 (socket says 172.21.15.120:50060) 2024-02-06T22:18:01.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.539+0000 7fd401d9b640 1 -- 172.21.15.120:0/4255796575 learned_addr learned my addr 172.21.15.120:0/4255796575 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T22:18:01.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.539+0000 7fd40259c640 1 --2- 172.21.15.120:0/4255796575 >> v2:172.21.15.120:3300/0 conn(0x7fd4040af4b0 0x7fd40413b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:01.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.540+0000 7fd40259c640 10 monclient(hunting): get_auth_request con 0x7fd4040af4b0 auth_method 0 2024-02-06T22:18:01.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.540+0000 7fd40259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:01.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.540+0000 7fd40259c640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:01.541 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.540+0000 7fd40259c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:01.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.541+0000 7fd40259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:01.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.541+0000 7fd40259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:01.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.541+0000 7fd40259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:01.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.542+0000 7fd40159a640 1 -- 172.21.15.120:0/4255796575 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148134542 0 0) 0x7fd4040711d0 con 0x7fd404142030 2024-02-06T22:18:01.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.542+0000 7fd40159a640 10 monclient(hunting): _negotiate 2024-02-06T22:18:01.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.542+0000 7fd40159a640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:01.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.542+0000 7fd40159a640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:01.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.542+0000 7fd40159a640 10 monclient(hunting): my global_id is 21034 2024-02-06T22:18:01.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.542+0000 7fd40159a640 1 -- 172.21.15.120:0/4255796575 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3ec003520 con 0x7fd404142030 2024-02-06T22:18:01.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.542+0000 7fd40159a640 1 -- 172.21.15.120:0/4255796575 <== mon.0 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208731116 0 0) 0x7fd404070ef0 con 0x7fd404071eb0 2024-02-06T22:18:01.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.542+0000 7fd40159a640 10 monclient(hunting): _negotiate 2024-02-06T22:18:01.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.542+0000 7fd40159a640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:01.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.542+0000 7fd40159a640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:01.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.542+0000 7fd40159a640 10 monclient(hunting): my global_id is 16521 2024-02-06T22:18:01.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.542+0000 7fd40159a640 1 -- 172.21.15.120:0/4255796575 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4040711d0 con 0x7fd404071eb0 2024-02-06T22:18:01.544 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.544+0000 7fd40259c640 10 monclient(hunting): handle_auth_done global_id 16518 payload 274 2024-02-06T22:18:01.544 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.544+0000 7fd40259c640 1 -- 172.21.15.120:0/4255796575 >> v1:172.21.15.135:6789/0 conn(0x7fd404142030 legacy=0x7fd4041424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:01.544 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.544+0000 7fd40259c640 1 -- 172.21.15.120:0/4255796575 >> v1:172.21.15.120:6789/0 conn(0x7fd404071eb0 legacy=0x7fd40413be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:01.545 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.544+0000 7fd40259c640 10 monclient: _finish_hunting 0 2024-02-06T22:18:01.545 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.544+0000 7fd40259c640 1 monclient: found mon.noname-a 2024-02-06T22:18:01.545 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.544+0000 7fd40259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:01.545 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.544+0000 7fd40259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:18:01.545 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.544+0000 7fd40259c640 1 -- 172.21.15.120:0/4255796575 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4040714b0 con 0x7fd4040af4b0 2024-02-06T22:18:01.548 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.548+0000 7fd40259c640 1 --2- 172.21.15.120:0/4255796575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4040af4b0 0x7fd40413b8e0 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7fd3f400aea0 tx=0x7fd3f402cfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6a8224f962c0627 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:01.555 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd40159a640 1 -- 172.21.15.120:0/4255796575 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f402db60 con 0x7fd4040af4b0 2024-02-06T22:18:01.555 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd40159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:01.555 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd40159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:18:01.555 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd40159a640 10 monclient: dump: 2024-02-06T22:18:01.555 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:01.555 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:01.555 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.555 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.555 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:01.555 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:01.555 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:01.555 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:01.556 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:01.556 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:01.556 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd40159a640 10 monclient: _finish_auth 0 2024-02-06T22:18:01.556 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd40159a640 10 monclient: _check_auth_tickets 2024-02-06T22:18:01.556 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd40159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:01.556 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd40159a640 1 -- 172.21.15.120:0/4255796575 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f402dcc0 con 0x7fd4040af4b0 2024-02-06T22:18:01.556 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd40159a640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:01.556 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd40159a640 1 -- 172.21.15.120:0/4255796575 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4008660 con 0x7fd4040af4b0 2024-02-06T22:18:01.556 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd408932640 5 monclient: authenticate success, global_id 16518 2024-02-06T22:18:01.556 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd408932640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:01.556 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd408932640 4 set_mon_vals no callback set 2024-02-06T22:18:01.557 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd40159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:01.557 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd40159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:18:01.557 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.554+0000 7fd40159a640 10 monclient: dump: 2024-02-06T22:18:01.557 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:01.557 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:01.557 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.557 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.557 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:01.557 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:01.557 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:01.557 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:01.557 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:01.557 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:01.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.555+0000 7fd408932640 10 monclient: shutdown 2024-02-06T22:18:01.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.555+0000 7fd408932640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:01.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.555+0000 7fd408932640 1 -- 172.21.15.120:0/4255796575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4040af4b0 msgr2=0x7fd40413b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:01.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.555+0000 7fd408932640 1 --2- 172.21.15.120:0/4255796575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4040af4b0 0x7fd40413b8e0 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7fd3f400aea0 tx=0x7fd3f402cfc0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.555+0000 7fd408932640 1 -- 172.21.15.120:0/4255796575 shutdown_connections 2024-02-06T22:18:01.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.555+0000 7fd408932640 1 --2- 172.21.15.120:0/4255796575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4040af4b0 0x7fd40413b8e0 unknown :-1 s=CLOSED pgs=3347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.555+0000 7fd408932640 1 -- 172.21.15.120:0/4255796575 >> 172.21.15.120:0/4255796575 conn(0x7fd4040b3fb0 msgr2=0x7fd4040b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:01.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.555+0000 7fd408932640 1 -- 172.21.15.120:0/4255796575 shutdown_connections 2024-02-06T22:18:01.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.555+0000 7fd408932640 1 -- 172.21.15.120:0/4255796575 wait complete. 2024-02-06T22:18:01.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.556+0000 7fd408932640 10 monclient: build_initial_monmap 2024-02-06T22:18:01.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.556+0000 7fd408932640 10 monclient: monmap: 2024-02-06T22:18:01.561 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:01.562 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:01.562 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:01.562 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:01.562 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:01.562 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:01.562 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:01.562 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:01.562 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:01.562 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:01.562 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.556+0000 7fd408932640 1 librados: starting msgr at 2024-02-06T22:18:01.562 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.556+0000 7fd408932640 1 librados: starting objecter 2024-02-06T22:18:01.562 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.556+0000 7fd408932640 1 Processor -- start 2024-02-06T22:18:01.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.556+0000 7fd408932640 1 -- start start 2024-02-06T22:18:01.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.556+0000 7fd408932640 1 librados: setting wanted keys 2024-02-06T22:18:01.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.556+0000 7fd408932640 1 librados: calling monclient init 2024-02-06T22:18:01.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.556+0000 7fd408932640 10 monclient: init 2024-02-06T22:18:01.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:01.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T22:18:01.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd404071eb0 0x7fd4041376f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:01.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 10 monclient(hunting): picked mon.noname-b con 0x7fd404071eb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:01.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4040af4b0 0x7fd4041d5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:01.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 10 monclient(hunting): picked mon.noname-a con 0x7fd4040af4b0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:01.563 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd404142030 0x7fd4041d5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:01.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 10 monclient(hunting): picked mon.noname-c con 0x7fd404142030 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:01.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:01.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4041d9870 con 0x7fd4040af4b0 2024-02-06T22:18:01.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:01.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40406fcb0 con 0x7fd404071eb0 2024-02-06T22:18:01.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:01.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40413ca50 con 0x7fd404142030 2024-02-06T22:18:01.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:01.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd408932640 10 monclient(hunting): authenticate will time out at 3511.893743s 2024-02-06T22:18:01.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd401d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4040af4b0 0x7fd4041d5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:01.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd402d9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd404142030 0x7fd4041d5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:01.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd401d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4040af4b0 0x7fd4041d5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:41592/0 (socket says 172.21.15.120:41592) 2024-02-06T22:18:01.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd401d9b640 1 -- 172.21.15.120:0/2110070026 learned_addr learned my addr 172.21.15.120:0/2110070026 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:01.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd401d9b640 10 monclient(hunting): get_auth_request con 0x7fd4040af4b0 auth_method 0 2024-02-06T22:18:01.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd401d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:01.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd401d9b640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:01.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd401d9b640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:01.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.557+0000 7fd40259c640 1 --2- 172.21.15.120:0/2110070026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd404071eb0 0x7fd4041376f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:01.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd402d9d640 10 monclient(hunting): get_auth_request con 0x7fd404142030 auth_method 0 2024-02-06T22:18:01.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd402d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:01.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd402d9d640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:01.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd402d9d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:01.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd40259c640 10 monclient(hunting): get_auth_request con 0x7fd404071eb0 auth_method 0 2024-02-06T22:18:01.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd40259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:01.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd40259c640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:01.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd40259c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:01.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd401d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:01.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd401d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:01.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd401d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:01.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd402d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:01.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd402d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:01.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd402d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:01.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd40259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:01.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd40259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:01.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.558+0000 7fd40259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:01.568 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.561+0000 7fd401d9b640 10 monclient(hunting): handle_auth_done global_id 16524 payload 824 2024-02-06T22:18:01.568 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.561+0000 7fd401d9b640 1 -- 172.21.15.120:0/2110070026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd404142030 msgr2=0x7fd4041d5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:01.568 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.561+0000 7fd401d9b640 1 --2- 172.21.15.120:0/2110070026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd404142030 0x7fd4041d5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.568 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.561+0000 7fd401d9b640 1 -- 172.21.15.120:0/2110070026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd404071eb0 msgr2=0x7fd4041376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:01.568 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.561+0000 7fd401d9b640 1 --2- 172.21.15.120:0/2110070026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd404071eb0 0x7fd4041376f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.568 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.561+0000 7fd401d9b640 10 monclient: _finish_hunting 0 2024-02-06T22:18:01.568 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.561+0000 7fd401d9b640 1 monclient: found mon.noname-a 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.561+0000 7fd401d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.561+0000 7fd401d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.561+0000 7fd401d9b640 1 -- 172.21.15.120:0/2110070026 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f400ab80 con 0x7fd4040af4b0 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.563+0000 7fd401d9b640 1 --2- 172.21.15.120:0/2110070026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4040af4b0 0x7fd4041d5960 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800c400 tx=0x7fd3f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.563+0000 7fd3f2ffd640 10 client.?.objecter ms_handle_connect 0x7fd4040af4b0 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.563+0000 7fd3f2ffd640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd3f2ffd640 1 -- 172.21.15.120:0/2110070026 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8005ce0 con 0x7fd4040af4b0 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd3f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd3f2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd3f2ffd640 10 monclient: dump: 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:01.569 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:01.570 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:01.570 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:01.570 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:01.570 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:01.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd3f2ffd640 10 monclient: _finish_auth 0 2024-02-06T22:18:01.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd3f2ffd640 10 monclient: _check_auth_tickets 2024-02-06T22:18:01.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd3f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:01.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd3f2ffd640 1 -- 172.21.15.120:0/2110070026 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f8005e60 con 0x7fd4040af4b0 2024-02-06T22:18:01.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd408932640 5 monclient: authenticate success, global_id 16524 2024-02-06T22:18:01.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd408932640 10 monclient: _renew_subs 2024-02-06T22:18:01.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd408932640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:01.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd408932640 1 -- 172.21.15.120:0/2110070026 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4041d5440 con 0x7fd4040af4b0 2024-02-06T22:18:01.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd3f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:01.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd3f2ffd640 1 -- 172.21.15.120:0/2110070026 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f80072f0 con 0x7fd4040af4b0 2024-02-06T22:18:01.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd3f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:01.571 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd3f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:18:01.571 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd3f2ffd640 10 monclient: dump: 2024-02-06T22:18:01.571 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:01.571 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:01.571 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.571 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.571 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:01.571 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:01.571 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:01.572 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:01.572 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:01.572 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:01.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd400d99640 4 set_mon_vals no callback set 2024-02-06T22:18:01.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd408932640 10 client.16524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:01.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd408932640 10 monclient: _renew_subs 2024-02-06T22:18:01.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd408932640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:01.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd408932640 1 -- 172.21.15.120:0/2110070026 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4041d89a0 con 0x7fd4040af4b0 2024-02-06T22:18:01.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd408932640 1 librados: init done 2024-02-06T22:18:01.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd408932640 10 client.16524.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:01.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd408932640 15 client.16524.objecter _session_command_op_assign -1 1 2024-02-06T22:18:01.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd408932640 20 client.16524.objecter _get_session osd=-1 returning homeless 2024-02-06T22:18:01.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd408932640 10 client.16524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:01.573 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd408932640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:01.573 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.567+0000 7fd408932640 1 -- 172.21.15.120:0/2110070026 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd404142bf0 con 0x7fd4040af4b0 2024-02-06T22:18:01.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.586+0000 7fd3f2ffd640 1 -- 172.21.15.120:0/2110070026 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd3f8007450 con 0x7fd4040af4b0 2024-02-06T22:18:01.587 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.587+0000 7fd3f2ffd640 1 --2- 172.21.15.120:0/2110070026 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd3e406d1a0 0x7fd3e406f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:01.587 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.587+0000 7fd40259c640 1 --2- 172.21.15.120:0/2110070026 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd3e406d1a0 0x7fd3e406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:01.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.587+0000 7fd40259c640 10 monclient: get_auth_request con 0x7fd3e406d1a0 auth_method 0 2024-02-06T22:18:01.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.587+0000 7fd40259c640 1 --2- 172.21.15.120:0/2110070026 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd3e406d1a0 0x7fd3e406f660 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4036000 tx=0x7fd3f4031000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:01.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.587+0000 7fd3f2ffd640 10 client.16524.objecter ms_handle_connect 0x7fd3e406d1a0 2024-02-06T22:18:01.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 1 -- 172.21.15.120:0/2110070026 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fd3f8058730 con 0x7fd4040af4b0 2024-02-06T22:18:01.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 10 client.16524.objecter ms_dispatch 0x7fd404000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:01.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 3 client.16524.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:01.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 3 client.16524.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:01.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 10 client.16524.objecter checking command 1 2024-02-06T22:18:01.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 1 --2- 172.21.15.120:0/2110070026 >> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] conn(0x7fd3e4074f30 0x7fd3e40773d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:01.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 20 client.16524.objecter _get_session s=0x7fd3e4074860 osd=1 3 2024-02-06T22:18:01.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 20 client.16524.objecter put_session s=0x7fd3e4074860 osd=1 3 2024-02-06T22:18:01.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 15 client.16524.objecter _session_command_op_remove -1 1 2024-02-06T22:18:01.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 20 client.16524.objecter _get_session s=0x7fd3e4074860 osd=1 3 2024-02-06T22:18:01.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 20 client.16524.objecter get_session s=0x7fd3e4074860 osd=1 3 2024-02-06T22:18:01.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 15 client.16524.objecter _session_command_op_assign 1 1 2024-02-06T22:18:01.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 20 client.16524.objecter put_session s=0x7fd3e4074860 osd=1 4 2024-02-06T22:18:01.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 10 client.16524.objecter _send_command 1 2024-02-06T22:18:01.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 1 -- 172.21.15.120:0/2110070026 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3e4077aa0 con 0x7fd3e4074f30 2024-02-06T22:18:01.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 20 client.16524.objecter dump_active .. 0 homeless 2024-02-06T22:18:01.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 1 -- 172.21.15.120:0/2110070026 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_get_version_reply(handle=1 version=1346) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3f80549d0 con 0x7fd4040af4b0 2024-02-06T22:18:01.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.590+0000 7fd3f2ffd640 10 monclient: handle_get_version_reply finishing 1 version 1346 2024-02-06T22:18:01.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.591+0000 7fd402d9d640 1 --2- 172.21.15.120:0/2110070026 >> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] conn(0x7fd3e4074f30 0x7fd3e40773d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:01.593 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.591+0000 7fd402d9d640 10 monclient: get_auth_request con 0x7fd3e4074f30 auth_method 0 2024-02-06T22:18:01.594 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.594+0000 7fd402d9d640 1 --2- 172.21.15.120:0/2110070026 >> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] conn(0x7fd3e4074f30 0x7fd3e40773d0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:01.594 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.594+0000 7fd3f2ffd640 10 client.16524.objecter ms_handle_connect 0x7fd3e4074f30 2024-02-06T22:18:01.618 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.617+0000 7fd3f2ffd640 1 -- 172.21.15.120:0/2110070026 <== osd.1 v2:172.21.15.120:6808/3348315891 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd3e4077aa0 con 0x7fd3e4074f30 2024-02-06T22:18:01.618 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.618+0000 7fd3f2ffd640 10 client.16524.objecter ms_dispatch 0x7fd404000b90 command_reply(tid 1: 0 ) v1 2024-02-06T22:18:01.618 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.618+0000 7fd3f2ffd640 10 client.16524.objecter _finish_command 1 = system:0 2024-02-06T22:18:01.618 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.618+0000 7fd3f2ffd640 20 client.16524.objecter put_session s=0x7fd3e4074860 osd=1 4 2024-02-06T22:18:01.618 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.618+0000 7fd3f2ffd640 15 client.16524.objecter _session_command_op_remove 1 1 2024-02-06T22:18:01.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.632+0000 7fd408932640 10 client.16524.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-02-06T22:18:01.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.632+0000 7fd408932640 15 client.16524.objecter _session_command_op_assign -1 2 2024-02-06T22:18:01.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.632+0000 7fd408932640 20 client.16524.objecter _get_session s=0x7fd3e4074860 osd=1 3 2024-02-06T22:18:01.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.632+0000 7fd408932640 20 client.16524.objecter put_session s=0x7fd3e4074860 osd=1 3 2024-02-06T22:18:01.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.632+0000 7fd408932640 20 client.16524.objecter _get_session s=0x7fd3e4074860 osd=1 3 2024-02-06T22:18:01.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.632+0000 7fd408932640 15 client.16524.objecter _session_command_op_remove -1 2 2024-02-06T22:18:01.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.632+0000 7fd408932640 20 client.16524.objecter get_session s=0x7fd3e4074860 osd=1 3 2024-02-06T22:18:01.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.632+0000 7fd408932640 15 client.16524.objecter _session_command_op_assign 1 2 2024-02-06T22:18:01.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.632+0000 7fd408932640 20 client.16524.objecter put_session s=0x7fd3e4074860 osd=1 4 2024-02-06T22:18:01.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.632+0000 7fd408932640 10 client.16524.objecter _send_command 2 2024-02-06T22:18:01.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.632+0000 7fd408932640 1 -- 172.21.15.120:0/2110070026 --> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd4041d5680 con 0x7fd3e4074f30 2024-02-06T22:18:01.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd3f2ffd640 1 -- 172.21.15.120:0/2110070026 <== osd.1 v2:172.21.15.120:6808/3348315891 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd4041d5680 con 0x7fd3e4074f30 2024-02-06T22:18:01.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd3f2ffd640 10 client.16524.objecter ms_dispatch 0x7fd404000b90 command_reply(tid 2: 0 ) v1 2024-02-06T22:18:01.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd3f2ffd640 10 client.16524.objecter _finish_command 2 = system:0 2024-02-06T22:18:01.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd3f2ffd640 20 client.16524.objecter put_session s=0x7fd3e4074860 osd=1 4 2024-02-06T22:18:01.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd3f2ffd640 15 client.16524.objecter _session_command_op_remove 1 2 2024-02-06T22:18:01.646 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-02-06T22:18:01.646 INFO:teuthology.orchestra.run.smithi120.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-06T22:18:01.646 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-02-06T22:18:01.647 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 10 librados: watch_flush enter 2024-02-06T22:18:01.647 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 10 librados: watch_flush exit 2024-02-06T22:18:01.647 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 10 client.16524.objecter close_session for osd.1 2024-02-06T22:18:01.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 1 -- 172.21.15.120:0/2110070026 >> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] conn(0x7fd3e4074f30 msgr2=0x7fd3e40773d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:01.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 1 --2- 172.21.15.120:0/2110070026 >> [v2:172.21.15.120:6808/3348315891,v1:172.21.15.120:6809/3348315891] conn(0x7fd3e4074f30 0x7fd3e40773d0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 20 client.16524.objecter put_session s=0x7fd3e4074860 osd=1 1 2024-02-06T22:18:01.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 20 client.16524.objecter shutdown clearing up homeless session... 2024-02-06T22:18:01.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 10 client.16524.objecter successfully canceled tick 2024-02-06T22:18:01.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 1 -- 172.21.15.120:0/2110070026 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd3e406d1a0 msgr2=0x7fd3e406f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:01.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 1 --2- 172.21.15.120:0/2110070026 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fd3e406d1a0 0x7fd3e406f660 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4036000 tx=0x7fd3f4031000 comp rx=0 tx=0).stop 2024-02-06T22:18:01.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 10 monclient: shutdown 2024-02-06T22:18:01.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:01.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 1 -- 172.21.15.120:0/2110070026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4040af4b0 msgr2=0x7fd4041d5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:01.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 1 --2- 172.21.15.120:0/2110070026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4040af4b0 0x7fd4041d5960 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800c400 tx=0x7fd3f800c8c0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd402d9d640 1 -- 172.21.15.120:0/2110070026 reap_dead start 2024-02-06T22:18:01.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 1 -- 172.21.15.120:0/2110070026 shutdown_connections 2024-02-06T22:18:01.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 1 -- 172.21.15.120:0/2110070026 >> 172.21.15.120:0/2110070026 conn(0x7fd4040b3fb0 msgr2=0x7fd404134c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:01.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 1 -- 172.21.15.120:0/2110070026 shutdown_connections 2024-02-06T22:18:01.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 1 -- 172.21.15.120:0/2110070026 wait complete. 2024-02-06T22:18:01.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.645+0000 7fd408932640 1 librados: shutdown 2024-02-06T22:18:01.656 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-02-06T22:18:01.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.755+0000 7f419992b640 10 monclient: get_monmap_and_config 2024-02-06T22:18:01.756 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.755+0000 7f419992b640 10 monclient: build_initial_monmap 2024-02-06T22:18:01.756 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.755+0000 7f419992b640 10 monclient: monmap: 2024-02-06T22:18:01.756 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:01.756 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:01.756 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:01.755634+0000 2024-02-06T22:18:01.756 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:01.755634+0000 2024-02-06T22:18:01.756 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:01.756 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:01.756 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:01.756 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:01.756 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:01.756 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:01.756 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:01.757 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:01.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.755+0000 7f419992b640 1 Processor -- start 2024-02-06T22:18:01.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 1 -- start start 2024-02-06T22:18:01.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 10 monclient: init 2024-02-06T22:18:01.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:01.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-02-06T22:18:01.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f41940a9c80 0x7f41940aa060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:01.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 10 monclient(hunting): picked mon.noname-a con 0x7f41940a9c80 addr v2:172.21.15.120:3300/0 2024-02-06T22:18:01.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4194142bb0 0x7f4194142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:01.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 10 monclient(hunting): picked mon.noname-b con 0x7f4194142bb0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:01.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f419413f810 0x7f419413fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:01.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 10 monclient(hunting): picked mon.noname-c con 0x7f419413f810 addr v2:172.21.15.135:3300/0 2024-02-06T22:18:01.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:01.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4194143510 con 0x7f41940a9c80 2024-02-06T22:18:01.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:01.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41940aa5a0 con 0x7f419413f810 2024-02-06T22:18:01.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:01.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4194142720 con 0x7f4194142bb0 2024-02-06T22:18:01.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:01.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.756+0000 7f419992b640 10 monclient(hunting): authenticate will time out at 3512.093202s 2024-02-06T22:18:01.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f4192ffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f41940a9c80 0x7f41940aa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:01.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f41927fc640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4194142bb0 0x7f4194142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:01.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f41937fe640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f419413f810 0x7f419413fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:01.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f4192ffd640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f41940a9c80 0x7f41940aa060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:41600/0 (socket says 172.21.15.120:41600) 2024-02-06T22:18:01.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f4192ffd640 1 -- 172.21.15.120:0/451005342 learned_addr learned my addr 172.21.15.120:0/451005342 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:01.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f4192ffd640 10 monclient(hunting): get_auth_request con 0x7f41940a9c80 auth_method 0 2024-02-06T22:18:01.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f4192ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:01.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f4192ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:01.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f4192ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:01.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f41927fc640 10 monclient(hunting): get_auth_request con 0x7f4194142bb0 auth_method 0 2024-02-06T22:18:01.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f41927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:01.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f41927fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:01.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f41927fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:01.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f41937fe640 10 monclient(hunting): get_auth_request con 0x7f419413f810 auth_method 0 2024-02-06T22:18:01.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f41937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:01.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f41937fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:01.761 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.757+0000 7f41937fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:01.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.758+0000 7f4192ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:01.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.758+0000 7f4192ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:01.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.758+0000 7f4192ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:01.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.758+0000 7f41927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:01.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.758+0000 7f41927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:01.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.758+0000 7f41927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:01.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.758+0000 7f41937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:01.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.758+0000 7f41937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:01.762 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.758+0000 7f41937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:01.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.759+0000 7f4192ffd640 10 monclient(hunting): handle_auth_done global_id 16527 payload 274 2024-02-06T22:18:01.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.759+0000 7f4192ffd640 1 -- 172.21.15.120:0/451005342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4194142bb0 msgr2=0x7f4194142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:01.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.759+0000 7f4192ffd640 1 --2- 172.21.15.120:0/451005342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4194142bb0 0x7f4194142fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.759+0000 7f4192ffd640 1 -- 172.21.15.120:0/451005342 >> v2:172.21.15.135:3300/0 conn(0x7f419413f810 msgr2=0x7f419413fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:01.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.759+0000 7f4192ffd640 1 --2- 172.21.15.120:0/451005342 >> v2:172.21.15.135:3300/0 conn(0x7f419413f810 0x7f419413fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.759+0000 7f4192ffd640 10 monclient: _finish_hunting 0 2024-02-06T22:18:01.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.759+0000 7f4192ffd640 1 monclient: found mon.noname-a 2024-02-06T22:18:01.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.759+0000 7f4192ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:01.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.759+0000 7f4192ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:18:01.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.759+0000 7f4192ffd640 1 -- 172.21.15.120:0/451005342 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41940ab120 con 0x7f41940a9c80 2024-02-06T22:18:01.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.761+0000 7f4192ffd640 1 --2- 172.21.15.120:0/451005342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41940a9c80 0x7f41940aa060 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7f41940b2220 tx=0x7f418802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b5b9ebf0af4e57 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:01.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f4191ffb640 1 -- 172.21.15.120:0/451005342 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4188031870 con 0x7f41940a9c80 2024-02-06T22:18:01.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f4191ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:01.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f4191ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:18:01.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f4191ffb640 10 monclient: dump: 2024-02-06T22:18:01.767 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:01.767 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:01.767 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.767 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.767 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:01.767 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:01.767 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:01.767 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:01.767 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:01.767 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:01.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f4191ffb640 10 monclient: _finish_auth 0 2024-02-06T22:18:01.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f4191ffb640 10 monclient: _check_auth_tickets 2024-02-06T22:18:01.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f4191ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:01.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f4191ffb640 1 -- 172.21.15.120:0/451005342 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41880319d0 con 0x7f41940a9c80 2024-02-06T22:18:01.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f419992b640 5 monclient: authenticate success, global_id 16527 2024-02-06T22:18:01.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f419992b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T22:18:01.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f4191ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:01.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f419992b640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:01.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f4191ffb640 1 -- 172.21.15.120:0/451005342 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4188031d60 con 0x7f41940a9c80 2024-02-06T22:18:01.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f4191ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:01.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f419992b640 4 set_mon_vals no callback set 2024-02-06T22:18:01.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f4191ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:18:01.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.765+0000 7f4191ffb640 10 monclient: dump: 2024-02-06T22:18:01.768 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:01.768 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:01.768 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.769 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.769 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:01.769 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:01.769 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:01.769 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:01.769 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:01.769 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:01.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.766+0000 7f419992b640 10 monclient: shutdown 2024-02-06T22:18:01.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.766+0000 7f419992b640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:01.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.766+0000 7f419992b640 1 -- 172.21.15.120:0/451005342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41940a9c80 msgr2=0x7f41940aa060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:01.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.766+0000 7f419992b640 1 --2- 172.21.15.120:0/451005342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41940a9c80 0x7f41940aa060 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7f41940b2220 tx=0x7f418802b910 comp rx=0 tx=0).stop 2024-02-06T22:18:01.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.766+0000 7f419992b640 1 -- 172.21.15.120:0/451005342 shutdown_connections 2024-02-06T22:18:01.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.766+0000 7f419992b640 1 --2- 172.21.15.120:0/451005342 >> v2:172.21.15.135:3300/0 conn(0x7f419413f810 0x7f419413fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.766+0000 7f419992b640 1 --2- 172.21.15.120:0/451005342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4194142bb0 0x7f4194142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.766+0000 7f419992b640 1 --2- 172.21.15.120:0/451005342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41940a9c80 0x7f41940aa060 unknown :-1 s=CLOSED pgs=3349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.766+0000 7f419992b640 1 -- 172.21.15.120:0/451005342 >> 172.21.15.120:0/451005342 conn(0x7f41940b4ba0 msgr2=0x7f41940b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:01.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.766+0000 7f419992b640 1 -- 172.21.15.120:0/451005342 shutdown_connections 2024-02-06T22:18:01.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.766+0000 7f419992b640 1 -- 172.21.15.120:0/451005342 wait complete. 2024-02-06T22:18:01.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.767+0000 7f419992b640 10 monclient: build_initial_monmap 2024-02-06T22:18:01.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.767+0000 7f419992b640 10 monclient: monmap: 2024-02-06T22:18:01.773 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:01.773 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:01.773 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:01.773 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:01.773 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:01.774 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:01.774 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:01.774 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:01.774 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:01.774 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:01.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.767+0000 7f419992b640 1 librados: starting msgr at 2024-02-06T22:18:01.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.767+0000 7f419992b640 1 librados: starting objecter 2024-02-06T22:18:01.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.767+0000 7f419992b640 1 Processor -- start 2024-02-06T22:18:01.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.767+0000 7f419992b640 1 -- start start 2024-02-06T22:18:01.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.767+0000 7f419992b640 1 librados: setting wanted keys 2024-02-06T22:18:01.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.767+0000 7f419992b640 1 librados: calling monclient init 2024-02-06T22:18:01.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.767+0000 7f419992b640 10 monclient: init 2024-02-06T22:18:01.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:01.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T22:18:01.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41940a9c80 0x7f41941d8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:01.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 10 monclient(hunting): picked mon.noname-c con 0x7f41940a9c80 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:01.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f419413f810 0x7f41940b6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:01.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 10 monclient(hunting): picked mon.noname-b con 0x7f419413f810 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:01.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4194142bb0 0x7f41940b7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:01.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 10 monclient(hunting): picked mon.noname-a con 0x7f4194142bb0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:01.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:01.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41941d7e80 con 0x7f4194142bb0 2024-02-06T22:18:01.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:01.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41940ac110 con 0x7f419413f810 2024-02-06T22:18:01.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:01.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41940b5e20 con 0x7f41940a9c80 2024-02-06T22:18:01.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:01.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f419992b640 10 monclient(hunting): authenticate will time out at 3512.104704s 2024-02-06T22:18:01.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f41937fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4194142bb0 0x7f41940b7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:01.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f41927fc640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f419413f810 0x7f41940b6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:01.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f4192ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41940a9c80 0x7f41941d8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:01.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f41937fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4194142bb0 0x7f41940b7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:41602/0 (socket says 172.21.15.120:41602) 2024-02-06T22:18:01.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f41937fe640 1 -- 172.21.15.120:0/949043049 learned_addr learned my addr 172.21.15.120:0/949043049 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:01.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f41937fe640 10 monclient(hunting): get_auth_request con 0x7f4194142bb0 auth_method 0 2024-02-06T22:18:01.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f41937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:01.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f41937fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:01.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.768+0000 7f41937fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:01.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f41927fc640 10 monclient(hunting): get_auth_request con 0x7f419413f810 auth_method 0 2024-02-06T22:18:01.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f41927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:01.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f41927fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:01.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f41927fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:01.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f4192ffd640 10 monclient(hunting): get_auth_request con 0x7f41940a9c80 auth_method 0 2024-02-06T22:18:01.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f4192ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:01.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f4192ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:01.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f4192ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:01.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f41937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:01.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f41937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:01.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f41937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:01.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f41927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:01.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f41927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:01.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f41927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:01.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f4192ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:01.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f4192ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:01.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.769+0000 7f4192ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:01.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.772+0000 7f41927fc640 10 monclient(hunting): handle_auth_done global_id 21043 payload 824 2024-02-06T22:18:01.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.772+0000 7f41927fc640 1 -- 172.21.15.120:0/949043049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41940a9c80 msgr2=0x7f41941d8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:01.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.772+0000 7f41927fc640 1 --2- 172.21.15.120:0/949043049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41940a9c80 0x7f41941d8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.777 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.772+0000 7f41927fc640 1 -- 172.21.15.120:0/949043049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4194142bb0 msgr2=0x7f41940b7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:01.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.772+0000 7f41927fc640 1 --2- 172.21.15.120:0/949043049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4194142bb0 0x7f41940b7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:01.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.772+0000 7f41927fc640 10 monclient: _finish_hunting 0 2024-02-06T22:18:01.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.772+0000 7f41927fc640 1 monclient: found mon.noname-b 2024-02-06T22:18:01.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.772+0000 7f41927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:01.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.772+0000 7f41927fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T22:18:01.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.772+0000 7f41927fc640 1 -- 172.21.15.120:0/949043049 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41880094d0 con 0x7f419413f810 2024-02-06T22:18:01.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.774+0000 7f41927fc640 1 --2- 172.21.15.120:0/949043049 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f419413f810 0x7f41940b6b30 secure :-1 s=READY pgs=5090 cs=0 l=1 rev1=1 crypto rx=0x7f418400e870 tx=0x7f418400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:01.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.774+0000 7f417bfff640 10 client.?.objecter ms_handle_connect 0x7f419413f810 2024-02-06T22:18:01.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.774+0000 7f417bfff640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:01.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f417bfff640 1 -- 172.21.15.120:0/949043049 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418400ee60 con 0x7f419413f810 2024-02-06T22:18:01.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f417bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:01.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f417bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:18:01.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f417bfff640 10 monclient: dump: 2024-02-06T22:18:01.780 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:01.780 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:01.780 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.780 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.780 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:01.780 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:01.780 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:01.780 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:01.781 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:01.781 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:01.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f417bfff640 10 monclient: _finish_auth 0 2024-02-06T22:18:01.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f417bfff640 10 monclient: _check_auth_tickets 2024-02-06T22:18:01.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f417bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:01.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f417bfff640 1 -- 172.21.15.120:0/949043049 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418400c940 con 0x7f419413f810 2024-02-06T22:18:01.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f419992b640 5 monclient: authenticate success, global_id 21043 2024-02-06T22:18:01.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f419992b640 10 monclient: _renew_subs 2024-02-06T22:18:01.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f419992b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:01.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f419992b640 1 -- 172.21.15.120:0/949043049 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41940b6040 con 0x7f419413f810 2024-02-06T22:18:01.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f417bfff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:01.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f417bfff640 1 -- 172.21.15.120:0/949043049 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418400ccd0 con 0x7f419413f810 2024-02-06T22:18:01.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f417bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:01.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f417bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T22:18:01.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f417bfff640 10 monclient: dump: 2024-02-06T22:18:01.782 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:01.782 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:01.782 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.782 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:01.782 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:01.782 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:01.782 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:01.782 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:01.782 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:01.782 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:01.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f419992b640 10 client.21043.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:01.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f419992b640 10 monclient: _renew_subs 2024-02-06T22:18:01.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f419992b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:01.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f419992b640 1 -- 172.21.15.120:0/949043049 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4194143ad0 con 0x7f419413f810 2024-02-06T22:18:01.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f419992b640 1 librados: init done 2024-02-06T22:18:01.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.778+0000 7f41917fa640 4 set_mon_vals no callback set 2024-02-06T22:18:01.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.779+0000 7f419992b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:01.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.779+0000 7f419992b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:01.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.779+0000 7f419992b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:01.783 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.779+0000 7f419992b640 1 -- 172.21.15.120:0/949043049 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4194068910 con 0x7f419413f810 2024-02-06T22:18:01.800 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.799+0000 7f417bfff640 1 -- 172.21.15.120:0/949043049 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4184013470 con 0x7f419413f810 2024-02-06T22:18:01.800 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.800+0000 7f417bfff640 1 --2- 172.21.15.120:0/949043049 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f418006d1f0 0x7f418006f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:01.800 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.800+0000 7f4192ffd640 1 --2- 172.21.15.120:0/949043049 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f418006d1f0 0x7f418006f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:01.800 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.800+0000 7f4192ffd640 10 monclient: get_auth_request con 0x7f418006d1f0 auth_method 0 2024-02-06T22:18:01.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.800+0000 7f4192ffd640 1 --2- 172.21.15.120:0/949043049 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f418006d1f0 0x7f418006f6b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4188036000 tx=0x7f418803c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:01.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:01.800+0000 7f417bfff640 10 client.21043.objecter ms_handle_connect 0x7f418006d1f0 2024-02-06T22:18:02.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.027+0000 7f417bfff640 1 -- 172.21.15.120:0/949043049 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f418400ce30 con 0x7f419413f810 2024-02-06T22:18:02.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.027+0000 7f417bfff640 10 client.21043.objecter ms_dispatch 0x7f4194000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:02.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.027+0000 7f417bfff640 3 client.21043.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:02.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.027+0000 7f417bfff640 3 client.21043.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:02.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.027+0000 7f417bfff640 20 client.21043.objecter dump_active .. 0 homeless 2024-02-06T22:18:02.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.037+0000 7f417bfff640 1 -- 172.21.15.120:0/949043049 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4184054fd0 con 0x7f419413f810 2024-02-06T22:18:02.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.037+0000 7f417bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:02.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.037+0000 7f417bfff640 10 monclient: _finish_command 1 = system:0 2024-02-06T22:18:02.231 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.231+0000 7f419992b640 1 -- 172.21.15.120:0/949043049 --> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f41940ab670 con 0x7f418006d1f0 2024-02-06T22:18:02.232 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.231+0000 7f417bfff640 1 -- 172.21.15.120:0/949043049 <== mgr.4100 v2:172.21.15.135:6800/315664144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4188031670 con 0x7f418006d1f0 2024-02-06T22:18:02.232 INFO:teuthology.orchestra.run.smithi120.stderr:instructed osd(s) 1 to deep-scrub 2024-02-06T22:18:02.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 10 librados: watch_flush enter 2024-02-06T22:18:02.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 10 librados: watch_flush exit 2024-02-06T22:18:02.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 20 client.21043.objecter shutdown clearing up homeless session... 2024-02-06T22:18:02.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 10 client.21043.objecter successfully canceled tick 2024-02-06T22:18:02.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 1 -- 172.21.15.120:0/949043049 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f418006d1f0 msgr2=0x7f418006f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:02.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 1 --2- 172.21.15.120:0/949043049 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f418006d1f0 0x7f418006f6b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4188036000 tx=0x7f418803c040 comp rx=0 tx=0).stop 2024-02-06T22:18:02.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 10 monclient: shutdown 2024-02-06T22:18:02.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:02.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 1 -- 172.21.15.120:0/949043049 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f419413f810 msgr2=0x7f41940b6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:02.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 1 --2- 172.21.15.120:0/949043049 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f419413f810 0x7f41940b6b30 secure :-1 s=READY pgs=5090 cs=0 l=1 rev1=1 crypto rx=0x7f418400e870 tx=0x7f418400ed30 comp rx=0 tx=0).stop 2024-02-06T22:18:02.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 1 -- 172.21.15.120:0/949043049 shutdown_connections 2024-02-06T22:18:02.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 1 --2- 172.21.15.120:0/949043049 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f418006d1f0 0x7f418006f6b0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.235 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 1 --2- 172.21.15.120:0/949043049 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f419413f810 0x7f41940b6b30 unknown :-1 s=CLOSED pgs=5090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.236 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 1 --2- 172.21.15.120:0/949043049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4194142bb0 0x7f41940b7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.236 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 1 --2- 172.21.15.120:0/949043049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41940a9c80 0x7f41941d8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.236 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.233+0000 7f419992b640 1 -- 172.21.15.120:0/949043049 >> 172.21.15.120:0/949043049 conn(0x7f41940b4ba0 msgr2=0x7f41940b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:02.236 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.234+0000 7f419992b640 1 -- 172.21.15.120:0/949043049 shutdown_connections 2024-02-06T22:18:02.236 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.234+0000 7f419992b640 1 -- 172.21.15.120:0/949043049 wait complete. 2024-02-06T22:18:02.236 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.234+0000 7f419992b640 1 librados: shutdown 2024-02-06T22:18:02.245 INFO:tasks.ceph:Scrubbing osd.2 2024-02-06T22:18:02.246 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-02-06T22:18:02.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.352+0000 7fb2154d9640 10 monclient: get_monmap_and_config 2024-02-06T22:18:02.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.352+0000 7fb2154d9640 10 monclient: build_initial_monmap 2024-02-06T22:18:02.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.352+0000 7fb2154d9640 10 monclient: monmap: 2024-02-06T22:18:02.353 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:02.353 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:02.353 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:02.352691+0000 2024-02-06T22:18:02.353 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:02.352691+0000 2024-02-06T22:18:02.353 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:02.353 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:02.353 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:02.353 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:02.353 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:02.353 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:02.353 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:02.353 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:02.355 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 1 Processor -- start 2024-02-06T22:18:02.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 1 -- start start 2024-02-06T22:18:02.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 10 monclient: init 2024-02-06T22:18:02.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:02.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-02-06T22:18:02.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2100b5240 0x7fb210142fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:02.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 10 monclient(hunting): picked mon.noname-b con 0x7fb2100b5240 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:02.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2100b5620 addr v1:172.21.15.120:6789/0 2024-02-06T22:18:02.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb2100aef10 0x7fb2100b0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:02.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 10 monclient(hunting): picked mon.noname-a con 0x7fb2100aef10 addr v2:172.21.15.120:3300/0 2024-02-06T22:18:02.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb210142760 con 0x7fb2100b5620 2024-02-06T22:18:02.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:02.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2100b2240 con 0x7fb2100aef10 2024-02-06T22:18:02.356 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2100b13a0 con 0x7fb2100b5240 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.353+0000 7fb2154d9640 10 monclient(hunting): authenticate will time out at 3512.690200s 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.354+0000 7fb20effd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2100b5240 0x7fb210142fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.354+0000 7fb20f7fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb2100aef10 0x7fb2100b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.354+0000 7fb20e7fc640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fb2100b5620 0x7fb2100afa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:50066/0 (socket says 172.21.15.120:50066) 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.354+0000 7fb20e7fc640 1 -- 172.21.15.120:0/1908158542 learned_addr learned my addr 172.21.15.120:0/1908158542 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.354+0000 7fb20effd640 10 monclient(hunting): get_auth_request con 0x7fb2100b5240 auth_method 0 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.354+0000 7fb20effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.354+0000 7fb20effd640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.354+0000 7fb20effd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.354+0000 7fb20f7fe640 10 monclient(hunting): get_auth_request con 0x7fb2100aef10 auth_method 0 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.354+0000 7fb20f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.354+0000 7fb20f7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.354+0000 7fb20f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:02.357 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.355+0000 7fb20effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:02.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.355+0000 7fb20effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:02.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.355+0000 7fb20effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:02.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.355+0000 7fb20f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:02.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.355+0000 7fb20f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:02.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.355+0000 7fb20f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:02.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.356+0000 7fb20effd640 10 monclient(hunting): handle_auth_done global_id 18710 payload 274 2024-02-06T22:18:02.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.356+0000 7fb20effd640 1 -- 172.21.15.120:0/1908158542 >> v2:172.21.15.120:3300/0 conn(0x7fb2100aef10 msgr2=0x7fb2100b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:02.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.356+0000 7fb20effd640 1 --2- 172.21.15.120:0/1908158542 >> v2:172.21.15.120:3300/0 conn(0x7fb2100aef10 0x7fb2100b0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.356+0000 7fb20effd640 1 -- 172.21.15.120:0/1908158542 >> v1:172.21.15.120:6789/0 conn(0x7fb2100b5620 legacy=0x7fb2100afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:02.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.356+0000 7fb20effd640 10 monclient: _finish_hunting 0 2024-02-06T22:18:02.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.356+0000 7fb20effd640 1 monclient: found mon.noname-b 2024-02-06T22:18:02.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.356+0000 7fb20effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:02.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.356+0000 7fb20effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T22:18:02.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.356+0000 7fb20effd640 1 -- 172.21.15.120:0/1908158542 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2100b15a0 con 0x7fb2100b5240 2024-02-06T22:18:02.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.358+0000 7fb20effd640 1 --2- 172.21.15.120:0/1908158542 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2100b5240 0x7fb210142fb0 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 crypto rx=0x7fb2040097a0 tx=0x7fb20402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d216bd49dd72fd61 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:02.363 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb20dffb640 1 -- 172.21.15.120:0/1908158542 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb204031830 con 0x7fb2100b5240 2024-02-06T22:18:02.363 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb20dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:02.363 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb20dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:18:02.363 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb20dffb640 10 monclient: dump: 2024-02-06T22:18:02.363 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:02.363 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:02.363 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.363 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.363 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:02.363 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:02.363 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:02.364 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:02.364 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:02.364 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:02.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb20dffb640 10 monclient: _finish_auth 0 2024-02-06T22:18:02.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb20dffb640 10 monclient: _check_auth_tickets 2024-02-06T22:18:02.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb20dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:02.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb20dffb640 1 -- 172.21.15.120:0/1908158542 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb204031990 con 0x7fb2100b5240 2024-02-06T22:18:02.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb2154d9640 5 monclient: authenticate success, global_id 18710 2024-02-06T22:18:02.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb2154d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T22:18:02.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb20dffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:02.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb2154d9640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:02.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb20dffb640 1 -- 172.21.15.120:0/1908158542 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb204031d20 con 0x7fb2100b5240 2024-02-06T22:18:02.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb20dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:02.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb2154d9640 4 set_mon_vals no callback set 2024-02-06T22:18:02.365 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb20dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T22:18:02.365 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.362+0000 7fb20dffb640 10 monclient: dump: 2024-02-06T22:18:02.365 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:02.365 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:02.365 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.365 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.365 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:02.365 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:02.365 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:02.365 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:02.365 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:02.365 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:02.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.363+0000 7fb2154d9640 10 monclient: shutdown 2024-02-06T22:18:02.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.363+0000 7fb2154d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:02.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.363+0000 7fb2154d9640 1 -- 172.21.15.120:0/1908158542 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2100b5240 msgr2=0x7fb210142fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:02.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.363+0000 7fb2154d9640 1 --2- 172.21.15.120:0/1908158542 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2100b5240 0x7fb210142fb0 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 crypto rx=0x7fb2040097a0 tx=0x7fb20402b8d0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.363+0000 7fb2154d9640 1 -- 172.21.15.120:0/1908158542 shutdown_connections 2024-02-06T22:18:02.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.363+0000 7fb2154d9640 1 --2- 172.21.15.120:0/1908158542 >> v2:172.21.15.120:3300/0 conn(0x7fb2100aef10 0x7fb2100b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.363+0000 7fb2154d9640 1 --2- 172.21.15.120:0/1908158542 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2100b5240 0x7fb210142fb0 unknown :-1 s=CLOSED pgs=3345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.369 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.363+0000 7fb2154d9640 1 -- 172.21.15.120:0/1908158542 >> 172.21.15.120:0/1908158542 conn(0x7fb2100b8660 msgr2=0x7fb2100b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:02.370 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.363+0000 7fb2154d9640 1 -- 172.21.15.120:0/1908158542 shutdown_connections 2024-02-06T22:18:02.370 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.363+0000 7fb2154d9640 1 -- 172.21.15.120:0/1908158542 wait complete. 2024-02-06T22:18:02.370 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.364+0000 7fb2154d9640 10 monclient: build_initial_monmap 2024-02-06T22:18:02.370 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.364+0000 7fb2154d9640 10 monclient: monmap: 2024-02-06T22:18:02.370 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:02.370 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:02.370 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:02.370 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:02.370 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:02.370 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:02.370 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:02.370 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:02.370 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.365+0000 7fb2154d9640 1 librados: starting msgr at 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.365+0000 7fb2154d9640 1 librados: starting objecter 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.365+0000 7fb2154d9640 1 Processor -- start 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.365+0000 7fb2154d9640 1 -- start start 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.365+0000 7fb2154d9640 1 librados: setting wanted keys 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.365+0000 7fb2154d9640 1 librados: calling monclient init 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.365+0000 7fb2154d9640 10 monclient: init 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2100aef10 0x7fb21013d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 10 monclient(hunting): picked mon.noname-a con 0x7fb2100aef10 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2100b5240 0x7fb2101dbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 10 monclient(hunting): picked mon.noname-b con 0x7fb2100b5240 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb21013df80 0x7fb2101dc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:02.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 10 monclient(hunting): picked mon.noname-c con 0x7fb21013df80 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:02.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:02.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21013f2b0 con 0x7fb2100aef10 2024-02-06T22:18:02.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:02.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21013efd0 con 0x7fb2100b5240 2024-02-06T22:18:02.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:02.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21013e4a0 con 0x7fb21013df80 2024-02-06T22:18:02.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:02.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb2154d9640 10 monclient(hunting): authenticate will time out at 3512.702822s 2024-02-06T22:18:02.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb20effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2100aef10 0x7fb21013d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:02.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb20f7fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb21013df80 0x7fb2101dc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:02.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.366+0000 7fb20e7fc640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2100b5240 0x7fb2101dbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:02.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2100aef10 0x7fb21013d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:41618/0 (socket says 172.21.15.120:41618) 2024-02-06T22:18:02.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20effd640 1 -- 172.21.15.120:0/1273532996 learned_addr learned my addr 172.21.15.120:0/1273532996 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:02.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20effd640 10 monclient(hunting): get_auth_request con 0x7fb2100aef10 auth_method 0 2024-02-06T22:18:02.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:02.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20effd640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:02.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20effd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:02.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20e7fc640 10 monclient(hunting): get_auth_request con 0x7fb2100b5240 auth_method 0 2024-02-06T22:18:02.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:02.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20e7fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:02.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:02.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20f7fe640 10 monclient(hunting): get_auth_request con 0x7fb21013df80 auth_method 0 2024-02-06T22:18:02.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:02.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20f7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:02.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:02.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:02.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:02.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:02.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:02.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:02.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:02.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:02.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:02.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.367+0000 7fb20f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:02.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.371+0000 7fb20e7fc640 10 monclient(hunting): handle_auth_done global_id 21046 payload 824 2024-02-06T22:18:02.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.371+0000 7fb20e7fc640 1 -- 172.21.15.120:0/1273532996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb21013df80 msgr2=0x7fb2101dc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:02.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.371+0000 7fb20e7fc640 1 --2- 172.21.15.120:0/1273532996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb21013df80 0x7fb2101dc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.371+0000 7fb20e7fc640 1 -- 172.21.15.120:0/1273532996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2100aef10 msgr2=0x7fb21013d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:02.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.371+0000 7fb20e7fc640 1 --2- 172.21.15.120:0/1273532996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2100aef10 0x7fb21013d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.371+0000 7fb20e7fc640 10 monclient: _finish_hunting 0 2024-02-06T22:18:02.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.371+0000 7fb20e7fc640 1 monclient: found mon.noname-b 2024-02-06T22:18:02.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.371+0000 7fb20e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:02.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.371+0000 7fb20e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T22:18:02.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.371+0000 7fb20e7fc640 1 -- 172.21.15.120:0/1273532996 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb204009400 con 0x7fb2100b5240 2024-02-06T22:18:02.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.373+0000 7fb20e7fc640 1 --2- 172.21.15.120:0/1273532996 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2100b5240 0x7fb2101dbeb0 secure :-1 s=READY pgs=5091 cs=0 l=1 rev1=1 crypto rx=0x7fb200000f60 tx=0x7fb20000c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:02.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.373+0000 7fb1f77fe640 10 client.?.objecter ms_handle_connect 0x7fb2100b5240 2024-02-06T22:18:02.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.373+0000 7fb1f77fe640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:02.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb1f77fe640 1 -- 172.21.15.120:0/1273532996 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb200005ce0 con 0x7fb2100b5240 2024-02-06T22:18:02.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb1f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:02.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb1f77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:18:02.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb1f77fe640 10 monclient: dump: 2024-02-06T22:18:02.378 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:02.378 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:02.378 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.378 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.378 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:02.378 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:02.379 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:02.379 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:02.379 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:02.379 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:02.379 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb1f77fe640 10 monclient: _finish_auth 0 2024-02-06T22:18:02.379 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb1f77fe640 10 monclient: _check_auth_tickets 2024-02-06T22:18:02.379 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb1f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:02.379 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb1f77fe640 1 -- 172.21.15.120:0/1273532996 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb200005e40 con 0x7fb2100b5240 2024-02-06T22:18:02.379 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb1f77fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:02.379 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb1f77fe640 1 -- 172.21.15.120:0/1273532996 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb200007310 con 0x7fb2100b5240 2024-02-06T22:18:02.379 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb1f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:02.379 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb1f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T22:18:02.379 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb1f77fe640 10 monclient: dump: 2024-02-06T22:18:02.379 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:02.380 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:02.380 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.380 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.380 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:02.380 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:02.380 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:02.380 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:02.380 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:02.380 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:02.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb2154d9640 5 monclient: authenticate success, global_id 21046 2024-02-06T22:18:02.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb2154d9640 10 monclient: _renew_subs 2024-02-06T22:18:02.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb2154d9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:02.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb2154d9640 1 -- 172.21.15.120:0/1273532996 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb210143670 con 0x7fb2100b5240 2024-02-06T22:18:02.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb20d7fa640 4 set_mon_vals no callback set 2024-02-06T22:18:02.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb2154d9640 10 client.21046.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:02.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb2154d9640 10 monclient: _renew_subs 2024-02-06T22:18:02.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb2154d9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:02.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb2154d9640 1 -- 172.21.15.120:0/1273532996 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2101dc930 con 0x7fb2100b5240 2024-02-06T22:18:02.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.377+0000 7fb2154d9640 1 librados: init done 2024-02-06T22:18:02.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.378+0000 7fb2154d9640 10 client.21046.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:02.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.378+0000 7fb2154d9640 15 client.21046.objecter _session_command_op_assign -1 1 2024-02-06T22:18:02.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.378+0000 7fb2154d9640 20 client.21046.objecter _get_session osd=-1 returning homeless 2024-02-06T22:18:02.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.378+0000 7fb2154d9640 10 client.21046.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:02.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.378+0000 7fb2154d9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:02.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.378+0000 7fb2154d9640 1 -- 172.21.15.120:0/1273532996 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb2101434f0 con 0x7fb2100b5240 2024-02-06T22:18:02.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.397+0000 7fb1f77fe640 1 -- 172.21.15.120:0/1273532996 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb200007470 con 0x7fb2100b5240 2024-02-06T22:18:02.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.397+0000 7fb1f77fe640 1 --2- 172.21.15.120:0/1273532996 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fb1fc06d1a0 0x7fb1fc06f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:02.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.397+0000 7fb20effd640 1 --2- 172.21.15.120:0/1273532996 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fb1fc06d1a0 0x7fb1fc06f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:02.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.397+0000 7fb20effd640 10 monclient: get_auth_request con 0x7fb1fc06d1a0 auth_method 0 2024-02-06T22:18:02.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.398+0000 7fb20effd640 1 --2- 172.21.15.120:0/1273532996 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fb1fc06d1a0 0x7fb1fc06f660 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb20402b910 tx=0x7fb20403c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:02.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.398+0000 7fb1f77fe640 10 client.21046.objecter ms_handle_connect 0x7fb1fc06d1a0 2024-02-06T22:18:02.401 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 1 -- 172.21.15.120:0/1273532996 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fb200007720 con 0x7fb2100b5240 2024-02-06T22:18:02.401 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 10 client.21046.objecter ms_dispatch 0x7fb210000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:02.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 3 client.21046.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:02.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 3 client.21046.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:02.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 10 client.21046.objecter checking command 1 2024-02-06T22:18:02.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 1 --2- 172.21.15.120:0/1273532996 >> [v2:172.21.15.120:6824/3229495265,v1:172.21.15.120:6825/3229495265] conn(0x7fb1fc074f30 0x7fb1fc0773d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:02.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 20 client.21046.objecter _get_session s=0x7fb1fc074860 osd=2 3 2024-02-06T22:18:02.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 20 client.21046.objecter put_session s=0x7fb1fc074860 osd=2 3 2024-02-06T22:18:02.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 15 client.21046.objecter _session_command_op_remove -1 1 2024-02-06T22:18:02.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 20 client.21046.objecter _get_session s=0x7fb1fc074860 osd=2 3 2024-02-06T22:18:02.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 20 client.21046.objecter get_session s=0x7fb1fc074860 osd=2 3 2024-02-06T22:18:02.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 15 client.21046.objecter _session_command_op_assign 2 1 2024-02-06T22:18:02.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 20 client.21046.objecter put_session s=0x7fb1fc074860 osd=2 4 2024-02-06T22:18:02.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 10 client.21046.objecter _send_command 1 2024-02-06T22:18:02.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 1 -- 172.21.15.120:0/1273532996 --> [v2:172.21.15.120:6824/3229495265,v1:172.21.15.120:6825/3229495265] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb1fc077aa0 con 0x7fb1fc074f30 2024-02-06T22:18:02.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 20 client.21046.objecter dump_active .. 0 homeless 2024-02-06T22:18:02.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 1 -- 172.21.15.120:0/1273532996 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=1346) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2000547d0 con 0x7fb2100b5240 2024-02-06T22:18:02.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.400+0000 7fb1f77fe640 10 monclient: handle_get_version_reply finishing 1 version 1346 2024-02-06T22:18:02.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.401+0000 7fb20f7fe640 1 --2- 172.21.15.120:0/1273532996 >> [v2:172.21.15.120:6824/3229495265,v1:172.21.15.120:6825/3229495265] conn(0x7fb1fc074f30 0x7fb1fc0773d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:02.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.401+0000 7fb20f7fe640 10 monclient: get_auth_request con 0x7fb1fc074f30 auth_method 0 2024-02-06T22:18:02.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.404+0000 7fb20f7fe640 1 --2- 172.21.15.120:0/1273532996 >> [v2:172.21.15.120:6824/3229495265,v1:172.21.15.120:6825/3229495265] conn(0x7fb1fc074f30 0x7fb1fc0773d0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:02.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.404+0000 7fb1f77fe640 10 client.21046.objecter ms_handle_connect 0x7fb1fc074f30 2024-02-06T22:18:02.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.427+0000 7fb1f77fe640 1 -- 172.21.15.120:0/1273532996 <== osd.2 v2:172.21.15.120:6824/3229495265 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb1fc077aa0 con 0x7fb1fc074f30 2024-02-06T22:18:02.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.427+0000 7fb1f77fe640 10 client.21046.objecter ms_dispatch 0x7fb210000b90 command_reply(tid 1: 0 ) v1 2024-02-06T22:18:02.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.427+0000 7fb1f77fe640 10 client.21046.objecter _finish_command 1 = system:0 2024-02-06T22:18:02.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.427+0000 7fb1f77fe640 20 client.21046.objecter put_session s=0x7fb1fc074860 osd=2 4 2024-02-06T22:18:02.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.427+0000 7fb1f77fe640 15 client.21046.objecter _session_command_op_remove 2 1 2024-02-06T22:18:02.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.440+0000 7fb2154d9640 10 client.21046.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-02-06T22:18:02.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.440+0000 7fb2154d9640 15 client.21046.objecter _session_command_op_assign -1 2 2024-02-06T22:18:02.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.440+0000 7fb2154d9640 20 client.21046.objecter _get_session s=0x7fb1fc074860 osd=2 3 2024-02-06T22:18:02.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.440+0000 7fb2154d9640 20 client.21046.objecter put_session s=0x7fb1fc074860 osd=2 3 2024-02-06T22:18:02.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.440+0000 7fb2154d9640 20 client.21046.objecter _get_session s=0x7fb1fc074860 osd=2 3 2024-02-06T22:18:02.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.440+0000 7fb2154d9640 15 client.21046.objecter _session_command_op_remove -1 2 2024-02-06T22:18:02.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.440+0000 7fb2154d9640 20 client.21046.objecter get_session s=0x7fb1fc074860 osd=2 3 2024-02-06T22:18:02.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.440+0000 7fb2154d9640 15 client.21046.objecter _session_command_op_assign 2 2 2024-02-06T22:18:02.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.440+0000 7fb2154d9640 20 client.21046.objecter put_session s=0x7fb1fc074860 osd=2 4 2024-02-06T22:18:02.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.440+0000 7fb2154d9640 10 client.21046.objecter _send_command 2 2024-02-06T22:18:02.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.440+0000 7fb2154d9640 1 -- 172.21.15.120:0/1273532996 --> [v2:172.21.15.120:6824/3229495265,v1:172.21.15.120:6825/3229495265] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb2100b2430 con 0x7fb1fc074f30 2024-02-06T22:18:02.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.452+0000 7fb1f77fe640 1 -- 172.21.15.120:0/1273532996 <== osd.2 v2:172.21.15.120:6824/3229495265 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb20800f040 con 0x7fb1fc074f30 2024-02-06T22:18:02.453 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.452+0000 7fb1f77fe640 10 client.21046.objecter ms_dispatch 0x7fb210000b90 command_reply(tid 2: 0 ) v1 2024-02-06T22:18:02.453 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.452+0000 7fb1f77fe640 10 client.21046.objecter _finish_command 2 = system:0 2024-02-06T22:18:02.453 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.452+0000 7fb1f77fe640 20 client.21046.objecter put_session s=0x7fb1fc074860 osd=2 4 2024-02-06T22:18:02.453 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.452+0000 7fb1f77fe640 15 client.21046.objecter _session_command_op_remove 2 2 2024-02-06T22:18:02.453 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-02-06T22:18:02.453 INFO:teuthology.orchestra.run.smithi120.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-06T22:18:02.453 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-02-06T22:18:02.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.452+0000 7fb2154d9640 10 librados: watch_flush enter 2024-02-06T22:18:02.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.452+0000 7fb2154d9640 10 librados: watch_flush exit 2024-02-06T22:18:02.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.452+0000 7fb2154d9640 10 client.21046.objecter close_session for osd.2 2024-02-06T22:18:02.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.452+0000 7fb2154d9640 1 -- 172.21.15.120:0/1273532996 >> [v2:172.21.15.120:6824/3229495265,v1:172.21.15.120:6825/3229495265] conn(0x7fb1fc074f30 msgr2=0x7fb1fc0773d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:02.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.452+0000 7fb2154d9640 1 --2- 172.21.15.120:0/1273532996 >> [v2:172.21.15.120:6824/3229495265,v1:172.21.15.120:6825/3229495265] conn(0x7fb1fc074f30 0x7fb1fc0773d0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.452+0000 7fb2154d9640 20 client.21046.objecter put_session s=0x7fb1fc074860 osd=2 1 2024-02-06T22:18:02.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.452+0000 7fb2154d9640 20 client.21046.objecter shutdown clearing up homeless session... 2024-02-06T22:18:02.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.452+0000 7fb2154d9640 10 client.21046.objecter successfully canceled tick 2024-02-06T22:18:02.455 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.453+0000 7fb2154d9640 1 -- 172.21.15.120:0/1273532996 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fb1fc06d1a0 msgr2=0x7fb1fc06f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:02.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.453+0000 7fb2154d9640 1 --2- 172.21.15.120:0/1273532996 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fb1fc06d1a0 0x7fb1fc06f660 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb20402b910 tx=0x7fb20403c040 comp rx=0 tx=0).stop 2024-02-06T22:18:02.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.453+0000 7fb2154d9640 10 monclient: shutdown 2024-02-06T22:18:02.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.453+0000 7fb2154d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:02.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.453+0000 7fb2154d9640 1 -- 172.21.15.120:0/1273532996 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2100b5240 msgr2=0x7fb2101dbeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:02.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.453+0000 7fb2154d9640 1 --2- 172.21.15.120:0/1273532996 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2100b5240 0x7fb2101dbeb0 secure :-1 s=READY pgs=5091 cs=0 l=1 rev1=1 crypto rx=0x7fb200000f60 tx=0x7fb20000c760 comp rx=0 tx=0).stop 2024-02-06T22:18:02.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.453+0000 7fb20f7fe640 1 -- 172.21.15.120:0/1273532996 reap_dead start 2024-02-06T22:18:02.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.453+0000 7fb2154d9640 1 -- 172.21.15.120:0/1273532996 shutdown_connections 2024-02-06T22:18:02.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.453+0000 7fb2154d9640 1 -- 172.21.15.120:0/1273532996 >> 172.21.15.120:0/1273532996 conn(0x7fb2100b8660 msgr2=0x7fb2100744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:02.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.453+0000 7fb2154d9640 1 -- 172.21.15.120:0/1273532996 shutdown_connections 2024-02-06T22:18:02.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.453+0000 7fb2154d9640 1 -- 172.21.15.120:0/1273532996 wait complete. 2024-02-06T22:18:02.456 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.453+0000 7fb2154d9640 1 librados: shutdown 2024-02-06T22:18:02.464 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-02-06T22:18:02.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.564+0000 7f6d54ba8640 10 monclient: get_monmap_and_config 2024-02-06T22:18:02.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.564+0000 7f6d54ba8640 10 monclient: build_initial_monmap 2024-02-06T22:18:02.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.564+0000 7f6d54ba8640 10 monclient: monmap: 2024-02-06T22:18:02.565 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:02.565 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:02.565 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:02.565213+0000 2024-02-06T22:18:02.566 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:02.565213+0000 2024-02-06T22:18:02.566 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:02.566 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:02.566 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:02.566 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:02.566 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:02.566 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:02.566 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:02.566 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:02.568 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.565+0000 7f6d54ba8640 1 Processor -- start 2024-02-06T22:18:02.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.565+0000 7f6d54ba8640 1 -- start start 2024-02-06T22:18:02.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.565+0000 7f6d54ba8640 10 monclient: init 2024-02-06T22:18:02.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.566+0000 7f6d54ba8640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:02.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.566+0000 7f6d54ba8640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-02-06T22:18:02.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.566+0000 7f6d54ba8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d500b54b0 0x7f6d500a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:02.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.566+0000 7f6d54ba8640 10 monclient(hunting): picked mon.noname-b con 0x7f6d500b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:02.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.566+0000 7f6d54ba8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d5013f830 addr v1:172.21.15.120:6789/0 2024-02-06T22:18:02.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.566+0000 7f6d54ba8640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6d50142680 0x7f6d50142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:02.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.566+0000 7f6d54ba8640 10 monclient(hunting): picked mon.noname-c con 0x7f6d50142680 addr v2:172.21.15.135:3300/0 2024-02-06T22:18:02.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.566+0000 7f6d54ba8640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d500ab720 con 0x7f6d5013f830 2024-02-06T22:18:02.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.566+0000 7f6d54ba8640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:02.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.566+0000 7f6d54ba8640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d500ab520 con 0x7f6d50142680 2024-02-06T22:18:02.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.566+0000 7f6d54ba8640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:02.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.566+0000 7f6d54ba8640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d500b5890 con 0x7f6d500b54b0 2024-02-06T22:18:02.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.566+0000 7f6d54ba8640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:02.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.566+0000 7f6d54ba8640 10 monclient(hunting): authenticate will time out at 3512.902974s 2024-02-06T22:18:02.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.566+0000 7f6d4e59c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d500b54b0 0x7f6d500a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:02.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4ed9d640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6d50142680 0x7f6d50142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:02.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4e59c640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d500b54b0 0x7f6d500a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:55752/0 (socket says 172.21.15.120:55752) 2024-02-06T22:18:02.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4e59c640 1 -- 172.21.15.120:0/1765842743 learned_addr learned my addr 172.21.15.120:0/1765842743 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:02.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4e59c640 10 monclient(hunting): get_auth_request con 0x7f6d500b54b0 auth_method 0 2024-02-06T22:18:02.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:02.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4e59c640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:02.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4e59c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:02.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4ed9d640 10 monclient(hunting): get_auth_request con 0x7f6d50142680 auth_method 0 2024-02-06T22:18:02.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:02.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4ed9d640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:02.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:02.571 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:02.571 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:02.571 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:02.571 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:02.571 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:02.571 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.567+0000 7f6d4ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:02.571 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.569+0000 7f6d4e59c640 10 monclient(hunting): handle_auth_done global_id 18716 payload 274 2024-02-06T22:18:02.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.569+0000 7f6d4e59c640 1 -- 172.21.15.120:0/1765842743 >> v2:172.21.15.135:3300/0 conn(0x7f6d50142680 msgr2=0x7f6d50142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:02.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.569+0000 7f6d4e59c640 1 --2- 172.21.15.120:0/1765842743 >> v2:172.21.15.135:3300/0 conn(0x7f6d50142680 0x7f6d50142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.569+0000 7f6d4e59c640 1 -- 172.21.15.120:0/1765842743 >> v1:172.21.15.120:6789/0 conn(0x7f6d5013f830 legacy=0x7f6d5013fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:02.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.569+0000 7f6d4e59c640 10 monclient: _finish_hunting 0 2024-02-06T22:18:02.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.569+0000 7f6d4e59c640 1 monclient: found mon.noname-b 2024-02-06T22:18:02.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.569+0000 7f6d4e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:02.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.569+0000 7f6d4e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T22:18:02.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.569+0000 7f6d4e59c640 1 -- 172.21.15.120:0/1765842743 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d500b4800 con 0x7f6d500b54b0 2024-02-06T22:18:02.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.571+0000 7f6d4e59c640 1 --2- 172.21.15.120:0/1765842743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d500b54b0 0x7f6d500a9860 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 crypto rx=0x7f6d44009870 tx=0x7f6d4402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f69d2ab0b64ab944 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:02.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d4d59a640 1 -- 172.21.15.120:0/1765842743 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d44031640 con 0x7f6d500b54b0 2024-02-06T22:18:02.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:02.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d4d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:18:02.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d4d59a640 10 monclient: dump: 2024-02-06T22:18:02.576 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:02.576 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:02.576 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.576 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.576 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:02.576 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:02.576 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:02.576 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d4d59a640 10 monclient: _finish_auth 0 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d4d59a640 10 monclient: _check_auth_tickets 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d4d59a640 1 -- 172.21.15.120:0/1765842743 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d440317a0 con 0x7f6d500b54b0 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d54ba8640 5 monclient: authenticate success, global_id 18716 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d54ba8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d4d59a640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d54ba8640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d54ba8640 4 set_mon_vals no callback set 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d4d59a640 1 -- 172.21.15.120:0/1765842743 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d44031b30 con 0x7f6d500b54b0 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d4d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.574+0000 7f6d4d59a640 10 monclient: dump: 2024-02-06T22:18:02.577 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.575+0000 7f6d54ba8640 10 monclient: shutdown 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.575+0000 7f6d54ba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.575+0000 7f6d54ba8640 1 -- 172.21.15.120:0/1765842743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d500b54b0 msgr2=0x7f6d500a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.575+0000 7f6d54ba8640 1 --2- 172.21.15.120:0/1765842743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d500b54b0 0x7f6d500a9860 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 crypto rx=0x7f6d44009870 tx=0x7f6d4402b8f0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.575+0000 7f6d54ba8640 1 -- 172.21.15.120:0/1765842743 shutdown_connections 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.575+0000 7f6d54ba8640 1 --2- 172.21.15.120:0/1765842743 >> v2:172.21.15.135:3300/0 conn(0x7f6d50142680 0x7f6d50142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.575+0000 7f6d54ba8640 1 --2- 172.21.15.120:0/1765842743 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d500b54b0 0x7f6d500a9860 unknown :-1 s=CLOSED pgs=3346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.578 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.575+0000 7f6d54ba8640 1 -- 172.21.15.120:0/1765842743 >> 172.21.15.120:0/1765842743 conn(0x7f6d500b8660 msgr2=0x7f6d500b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:02.579 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.575+0000 7f6d54ba8640 1 -- 172.21.15.120:0/1765842743 shutdown_connections 2024-02-06T22:18:02.579 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.575+0000 7f6d54ba8640 1 -- 172.21.15.120:0/1765842743 wait complete. 2024-02-06T22:18:02.581 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.576+0000 7f6d54ba8640 10 monclient: build_initial_monmap 2024-02-06T22:18:02.581 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.576+0000 7f6d54ba8640 10 monclient: monmap: 2024-02-06T22:18:02.581 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:02.581 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:02.581 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:02.581 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:02.581 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:02.581 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:02.581 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:02.582 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:02.582 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:02.582 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:02.582 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.576+0000 7f6d54ba8640 1 librados: starting msgr at 2024-02-06T22:18:02.582 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.576+0000 7f6d54ba8640 1 librados: starting objecter 2024-02-06T22:18:02.582 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.576+0000 7f6d54ba8640 1 Processor -- start 2024-02-06T22:18:02.582 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.576+0000 7f6d54ba8640 1 -- start start 2024-02-06T22:18:02.582 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.576+0000 7f6d54ba8640 1 librados: setting wanted keys 2024-02-06T22:18:02.582 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.576+0000 7f6d54ba8640 1 librados: calling monclient init 2024-02-06T22:18:02.582 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.576+0000 7f6d54ba8640 10 monclient: init 2024-02-06T22:18:02.582 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:02.582 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T22:18:02.582 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d500b54b0 0x7f6d500b7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:02.582 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 10 monclient(hunting): picked mon.noname-b con 0x7f6d500b54b0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:02.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d5013f830 0x7f6d501df030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:02.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 10 monclient(hunting): picked mon.noname-c con 0x7f6d5013f830 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:02.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d50142680 0x7f6d50136340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:02.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 10 monclient(hunting): picked mon.noname-a con 0x7f6d50142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:02.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:02.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d50133e50 con 0x7f6d50142680 2024-02-06T22:18:02.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:02.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d501df990 con 0x7f6d500b54b0 2024-02-06T22:18:02.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:02.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d501dfe90 con 0x7f6d5013f830 2024-02-06T22:18:02.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:02.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.577+0000 7f6d54ba8640 10 monclient(hunting): authenticate will time out at 3512.914009s 2024-02-06T22:18:02.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.578+0000 7f6d4e59c640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d500b54b0 0x7f6d500b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:02.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.578+0000 7f6d4e59c640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d500b54b0 0x7f6d500b7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:51100/0 (socket says 172.21.15.120:51100) 2024-02-06T22:18:02.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.578+0000 7f6d4e59c640 1 -- 172.21.15.120:0/936724698 learned_addr learned my addr 172.21.15.120:0/936724698 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:02.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.578+0000 7f6d4e59c640 10 monclient(hunting): get_auth_request con 0x7f6d500b54b0 auth_method 0 2024-02-06T22:18:02.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.578+0000 7f6d4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:02.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.578+0000 7f6d4e59c640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:02.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.578+0000 7f6d4e59c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:02.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.578+0000 7f6d4dd9b640 1 --2- 172.21.15.120:0/936724698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d5013f830 0x7f6d501df030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:02.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.578+0000 7f6d4e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:02.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.578+0000 7f6d4e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:02.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.578+0000 7f6d4e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:02.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.579+0000 7f6d4dd9b640 10 monclient(hunting): get_auth_request con 0x7f6d5013f830 auth_method 0 2024-02-06T22:18:02.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.579+0000 7f6d4dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:02.585 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.579+0000 7f6d4dd9b640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:02.585 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.579+0000 7f6d4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:02.585 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.580+0000 7f6d4dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:02.585 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.580+0000 7f6d4dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:02.585 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.580+0000 7f6d4dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:02.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.581+0000 7f6d4e59c640 10 monclient(hunting): handle_auth_done global_id 21052 payload 824 2024-02-06T22:18:02.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.581+0000 7f6d4e59c640 1 -- 172.21.15.120:0/936724698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d5013f830 msgr2=0x7f6d501df030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:02.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.581+0000 7f6d4e59c640 1 --2- 172.21.15.120:0/936724698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d5013f830 0x7f6d501df030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.581+0000 7f6d4e59c640 1 -- 172.21.15.120:0/936724698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d50142680 msgr2=0x7f6d50136340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:02.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.581+0000 7f6d4e59c640 1 --2- 172.21.15.120:0/936724698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d50142680 0x7f6d50136340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:02.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.581+0000 7f6d4e59c640 10 monclient: _finish_hunting 0 2024-02-06T22:18:02.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.581+0000 7f6d4e59c640 1 monclient: found mon.noname-b 2024-02-06T22:18:02.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.581+0000 7f6d4e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:02.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.581+0000 7f6d4e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T22:18:02.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.581+0000 7f6d4e59c640 1 -- 172.21.15.120:0/936724698 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d440094d0 con 0x7f6d500b54b0 2024-02-06T22:18:02.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.583+0000 7f6d4e59c640 1 --2- 172.21.15.120:0/936724698 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d500b54b0 0x7f6d500b7180 secure :-1 s=READY pgs=5092 cs=0 l=1 rev1=1 crypto rx=0x7f6d44000c00 tx=0x7f6d440046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:02.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.583+0000 7f6d3effd640 10 client.?.objecter ms_handle_connect 0x7f6d500b54b0 2024-02-06T22:18:02.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.583+0000 7f6d3effd640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:02.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d3effd640 1 -- 172.21.15.120:0/936724698 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4403a8b0 con 0x7f6d500b54b0 2024-02-06T22:18:02.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:02.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d3effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:18:02.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d3effd640 10 monclient: dump: 2024-02-06T22:18:02.588 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:02.588 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:02.588 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.588 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.588 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:02.588 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:02.588 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:02.588 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d3effd640 10 monclient: _finish_auth 0 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d3effd640 10 monclient: _check_auth_tickets 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d3effd640 1 -- 172.21.15.120:0/936724698 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4403aa10 con 0x7f6d500b54b0 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d3effd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d3effd640 1 -- 172.21.15.120:0/936724698 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4403ada0 con 0x7f6d500b54b0 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d3effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d3effd640 10 monclient: dump: 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:02.589 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:02.590 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:02.590 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:02.590 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:02.590 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:02.590 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:02.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d54ba8640 5 monclient: authenticate success, global_id 21052 2024-02-06T22:18:02.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d54ba8640 10 monclient: _renew_subs 2024-02-06T22:18:02.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d54ba8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:02.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d54ba8640 1 -- 172.21.15.120:0/936724698 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d501df570 con 0x7f6d500b54b0 2024-02-06T22:18:02.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d4cd99640 4 set_mon_vals no callback set 2024-02-06T22:18:02.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d54ba8640 10 client.21052.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:02.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d54ba8640 10 monclient: _renew_subs 2024-02-06T22:18:02.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d54ba8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:02.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d54ba8640 1 -- 172.21.15.120:0/936724698 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d50136a60 con 0x7f6d500b54b0 2024-02-06T22:18:02.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.587+0000 7f6d54ba8640 1 librados: init done 2024-02-06T22:18:02.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.588+0000 7f6d54ba8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:02.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.588+0000 7f6d54ba8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:02.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.588+0000 7f6d54ba8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:02.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.588+0000 7f6d54ba8640 1 -- 172.21.15.120:0/936724698 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d501dfb00 con 0x7f6d500b54b0 2024-02-06T22:18:02.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.606+0000 7f6d3effd640 1 -- 172.21.15.120:0/936724698 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6d4402f050 con 0x7f6d500b54b0 2024-02-06T22:18:02.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.606+0000 7f6d3effd640 1 --2- 172.21.15.120:0/936724698 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f6d3006d1a0 0x7f6d3006f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:02.607 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.607+0000 7f6d4dd9b640 1 --2- 172.21.15.120:0/936724698 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f6d3006d1a0 0x7f6d3006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:02.607 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.607+0000 7f6d4dd9b640 10 monclient: get_auth_request con 0x7f6d3006d1a0 auth_method 0 2024-02-06T22:18:02.607 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.607+0000 7f6d4dd9b640 1 --2- 172.21.15.120:0/936724698 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f6d3006d1a0 0x7f6d3006f660 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6d50137bc0 tx=0x7f6d40009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:02.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.607+0000 7f6d3effd640 10 client.21052.objecter ms_handle_connect 0x7f6d3006d1a0 2024-02-06T22:18:02.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.610+0000 7f6d3effd640 1 -- 172.21.15.120:0/936724698 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f6d4407b250 con 0x7f6d500b54b0 2024-02-06T22:18:02.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.610+0000 7f6d3effd640 10 client.21052.objecter ms_dispatch 0x7f6d50000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:02.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.610+0000 7f6d3effd640 3 client.21052.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:02.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.610+0000 7f6d3effd640 3 client.21052.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:02.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.610+0000 7f6d3effd640 20 client.21052.objecter dump_active .. 0 homeless 2024-02-06T22:18:02.844 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.844+0000 7f6d3effd640 1 -- 172.21.15.120:0/936724698 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6d4407a920 con 0x7f6d500b54b0 2024-02-06T22:18:02.844 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.844+0000 7f6d3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:02.844 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:02.844+0000 7f6d3effd640 10 monclient: _finish_command 1 = system:0 2024-02-06T22:18:03.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.028+0000 7f6d54ba8640 1 -- 172.21.15.120:0/936724698 --> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d50133fc0 con 0x7f6d3006d1a0 2024-02-06T22:18:03.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.028+0000 7f6d3effd640 1 -- 172.21.15.120:0/936724698 <== mgr.4100 v2:172.21.15.135:6800/315664144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6d50133fc0 con 0x7f6d3006d1a0 2024-02-06T22:18:03.029 INFO:teuthology.orchestra.run.smithi120.stderr:instructed osd(s) 2 to deep-scrub 2024-02-06T22:18:03.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 10 librados: watch_flush enter 2024-02-06T22:18:03.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 10 librados: watch_flush exit 2024-02-06T22:18:03.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 20 client.21052.objecter shutdown clearing up homeless session... 2024-02-06T22:18:03.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 10 client.21052.objecter successfully canceled tick 2024-02-06T22:18:03.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 1 -- 172.21.15.120:0/936724698 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f6d3006d1a0 msgr2=0x7f6d3006f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:03.031 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 1 --2- 172.21.15.120:0/936724698 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f6d3006d1a0 0x7f6d3006f660 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6d50137bc0 tx=0x7f6d40009040 comp rx=0 tx=0).stop 2024-02-06T22:18:03.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 10 monclient: shutdown 2024-02-06T22:18:03.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:03.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 1 -- 172.21.15.120:0/936724698 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d500b54b0 msgr2=0x7f6d500b7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:03.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 1 --2- 172.21.15.120:0/936724698 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d500b54b0 0x7f6d500b7180 secure :-1 s=READY pgs=5092 cs=0 l=1 rev1=1 crypto rx=0x7f6d44000c00 tx=0x7f6d440046c0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 1 -- 172.21.15.120:0/936724698 shutdown_connections 2024-02-06T22:18:03.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 1 --2- 172.21.15.120:0/936724698 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f6d3006d1a0 0x7f6d3006f660 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 1 --2- 172.21.15.120:0/936724698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d50142680 0x7f6d50136340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 1 --2- 172.21.15.120:0/936724698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d5013f830 0x7f6d501df030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 1 --2- 172.21.15.120:0/936724698 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d500b54b0 0x7f6d500b7180 unknown :-1 s=CLOSED pgs=5092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 1 -- 172.21.15.120:0/936724698 >> 172.21.15.120:0/936724698 conn(0x7f6d500b8660 msgr2=0x7f6d500744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:03.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 1 -- 172.21.15.120:0/936724698 shutdown_connections 2024-02-06T22:18:03.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 1 -- 172.21.15.120:0/936724698 wait complete. 2024-02-06T22:18:03.032 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.030+0000 7f6d54ba8640 1 librados: shutdown 2024-02-06T22:18:03.046 INFO:tasks.ceph:Scrubbing osd.3 2024-02-06T22:18:03.046 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-02-06T22:18:03.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.158+0000 7f91f5274640 10 monclient: get_monmap_and_config 2024-02-06T22:18:03.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.158+0000 7f91f5274640 10 monclient: build_initial_monmap 2024-02-06T22:18:03.159 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.158+0000 7f91f5274640 10 monclient: monmap: 2024-02-06T22:18:03.159 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:03.160 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:03.160 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:03.159181+0000 2024-02-06T22:18:03.160 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:03.159181+0000 2024-02-06T22:18:03.160 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:03.160 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:03.160 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:03.160 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:03.160 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:03.160 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:03.161 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:03.161 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:03.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.159+0000 7f91f5274640 1 Processor -- start 2024-02-06T22:18:03.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.159+0000 7f91f5274640 1 -- start start 2024-02-06T22:18:03.164 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.159+0000 7f91f5274640 10 monclient: init 2024-02-06T22:18:03.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.160+0000 7f91f5274640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:03.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.160+0000 7f91f5274640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-02-06T22:18:03.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.160+0000 7f91f5274640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f91f00b5230 0x7f91f00a9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:03.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.160+0000 7f91f5274640 10 monclient(hunting): picked mon.noname-a con 0x7f91f00b5230 addr v2:172.21.15.120:3300/0 2024-02-06T22:18:03.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.160+0000 7f91f5274640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91f013f960 addr v1:172.21.15.120:6789/0 2024-02-06T22:18:03.165 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.160+0000 7f91f5274640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f91f0142d00 0x7f91f013fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:03.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.160+0000 7f91f5274640 10 monclient(hunting): picked mon.noname-c con 0x7f91f0142d00 addr v2:172.21.15.135:3300/0 2024-02-06T22:18:03.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.160+0000 7f91f5274640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f00a9d90 con 0x7f91f013f960 2024-02-06T22:18:03.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.160+0000 7f91f5274640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:03.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.160+0000 7f91f5274640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f00b5610 con 0x7f91f00b5230 2024-02-06T22:18:03.166 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.160+0000 7f91f5274640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:03.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.160+0000 7f91f5274640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f00ab3e0 con 0x7f91f0142d00 2024-02-06T22:18:03.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.160+0000 7f91f5274640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:03.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.160+0000 7f91f5274640 10 monclient(hunting): authenticate will time out at 3513.496856s 2024-02-06T22:18:03.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.161+0000 7f91ef7fe640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f91f0142d00 0x7f91f013fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:03.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.161+0000 7f91ef7fe640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f91f0142d00 0x7f91f013fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:51108/0 (socket says 172.21.15.120:51108) 2024-02-06T22:18:03.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.161+0000 7f91ef7fe640 1 -- 172.21.15.120:0/1491780880 learned_addr learned my addr 172.21.15.120:0/1491780880 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:03.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.161+0000 7f91ef7fe640 10 monclient(hunting): get_auth_request con 0x7f91f0142d00 auth_method 0 2024-02-06T22:18:03.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.161+0000 7f91ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:03.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.161+0000 7f91ef7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:03.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.161+0000 7f91ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:03.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.161+0000 7f91ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:03.167 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.161+0000 7f91ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:03.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.161+0000 7f91ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:03.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.163+0000 7f91ef7fe640 10 monclient(hunting): handle_auth_done global_id 21055 payload 274 2024-02-06T22:18:03.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.163+0000 7f91ef7fe640 1 -- 172.21.15.120:0/1491780880 >> v2:172.21.15.120:3300/0 conn(0x7f91f00b5230 msgr2=0x7f91f00a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:03.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.163+0000 7f91ef7fe640 1 --2- 172.21.15.120:0/1491780880 >> v2:172.21.15.120:3300/0 conn(0x7f91f00b5230 0x7f91f00a9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.163+0000 7f91ef7fe640 1 -- 172.21.15.120:0/1491780880 >> v1:172.21.15.120:6789/0 conn(0x7f91f013f960 legacy=0x7f91f01425f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:03.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.163+0000 7f91ef7fe640 10 monclient: _finish_hunting 0 2024-02-06T22:18:03.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.163+0000 7f91ef7fe640 1 monclient: found mon.noname-c 2024-02-06T22:18:03.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.163+0000 7f91ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:03.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.163+0000 7f91ef7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T22:18:03.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.163+0000 7f91ef7fe640 1 -- 172.21.15.120:0/1491780880 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f00ab6c0 con 0x7f91f0142d00 2024-02-06T22:18:03.176 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.165+0000 7f91ef7fe640 1 --2- 172.21.15.120:0/1491780880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f0142d00 0x7f91f013fdf0 secure :-1 s=READY pgs=5093 cs=0 l=1 rev1=1 crypto rx=0x7f91f00b5940 tx=0x7f91e002b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2b2a802f3a192f server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:03.176 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91edffb640 1 -- 172.21.15.120:0/1491780880 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0031750 con 0x7f91f0142d00 2024-02-06T22:18:03.177 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:03.177 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91edffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T22:18:03.177 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91edffb640 10 monclient: dump: 2024-02-06T22:18:03.177 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:03.177 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:03.177 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.177 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.177 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:03.177 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:03.178 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:03.178 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:03.178 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:03.178 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:03.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91edffb640 10 monclient: _finish_auth 0 2024-02-06T22:18:03.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91edffb640 10 monclient: _check_auth_tickets 2024-02-06T22:18:03.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:03.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91edffb640 1 -- 172.21.15.120:0/1491780880 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e00318b0 con 0x7f91f0142d00 2024-02-06T22:18:03.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91f5274640 5 monclient: authenticate success, global_id 21055 2024-02-06T22:18:03.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91f5274640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T22:18:03.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91edffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:03.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91edffb640 1 -- 172.21.15.120:0/1491780880 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0031c40 con 0x7f91f0142d00 2024-02-06T22:18:03.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91f5274640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:03.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91f5274640 4 set_mon_vals no callback set 2024-02-06T22:18:03.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:03.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T22:18:03.180 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91edffb640 10 monclient: dump: 2024-02-06T22:18:03.180 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:03.180 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:03.180 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.180 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.180 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:03.180 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:03.180 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:03.181 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:03.181 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:03.181 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:03.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91f5274640 10 monclient: shutdown 2024-02-06T22:18:03.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91f5274640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:03.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91f5274640 1 -- 172.21.15.120:0/1491780880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f0142d00 msgr2=0x7f91f013fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:03.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91f5274640 1 --2- 172.21.15.120:0/1491780880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f0142d00 0x7f91f013fdf0 secure :-1 s=READY pgs=5093 cs=0 l=1 rev1=1 crypto rx=0x7f91f00b5940 tx=0x7f91e002b7f0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91f5274640 1 -- 172.21.15.120:0/1491780880 shutdown_connections 2024-02-06T22:18:03.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91f5274640 1 --2- 172.21.15.120:0/1491780880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f0142d00 0x7f91f013fdf0 unknown :-1 s=CLOSED pgs=5093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.182 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91f5274640 1 --2- 172.21.15.120:0/1491780880 >> v2:172.21.15.120:3300/0 conn(0x7f91f00b5230 0x7f91f00a9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.182 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.169+0000 7f91f5274640 1 -- 172.21.15.120:0/1491780880 >> 172.21.15.120:0/1491780880 conn(0x7f91f006d1c0 msgr2=0x7f91f006d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:03.182 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.170+0000 7f91f5274640 1 -- 172.21.15.120:0/1491780880 shutdown_connections 2024-02-06T22:18:03.182 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.170+0000 7f91f5274640 1 -- 172.21.15.120:0/1491780880 wait complete. 2024-02-06T22:18:03.182 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.170+0000 7f91f5274640 10 monclient: build_initial_monmap 2024-02-06T22:18:03.182 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.170+0000 7f91f5274640 10 monclient: monmap: 2024-02-06T22:18:03.182 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:03.182 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:03.183 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:03.183 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:03.183 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:03.183 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:03.183 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:03.183 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:03.183 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:03.183 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:03.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.170+0000 7f91f5274640 1 librados: starting msgr at 2024-02-06T22:18:03.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.170+0000 7f91f5274640 1 librados: starting objecter 2024-02-06T22:18:03.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 1 Processor -- start 2024-02-06T22:18:03.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 1 -- start start 2024-02-06T22:18:03.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 1 librados: setting wanted keys 2024-02-06T22:18:03.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 1 librados: calling monclient init 2024-02-06T22:18:03.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 10 monclient: init 2024-02-06T22:18:03.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:03.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T22:18:03.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91f00b5230 0x7f91f0133300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:03.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 10 monclient(hunting): picked mon.noname-c con 0x7f91f00b5230 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:03.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f013f960 0x7f91f0133840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:03.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 10 monclient(hunting): picked mon.noname-b con 0x7f91f013f960 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:03.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91f0142d00 0x7f91f00acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:03.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 10 monclient(hunting): picked mon.noname-a con 0x7f91f0142d00 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:03.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:03.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00ac6a0 con 0x7f91f0142d00 2024-02-06T22:18:03.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:03.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f0133d80 con 0x7f91f013f960 2024-02-06T22:18:03.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:03.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f0133ef0 con 0x7f91f00b5230 2024-02-06T22:18:03.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:03.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.171+0000 7f91f5274640 10 monclient(hunting): authenticate will time out at 3513.508278s 2024-02-06T22:18:03.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.172+0000 7f91ee7fc640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f013f960 0x7f91f0133840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:03.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.172+0000 7f91eeffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91f00b5230 0x7f91f0133300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:03.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.172+0000 7f91ee7fc640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f013f960 0x7f91f0133840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:51124/0 (socket says 172.21.15.120:51124) 2024-02-06T22:18:03.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.172+0000 7f91ee7fc640 1 -- 172.21.15.120:0/4172900607 learned_addr learned my addr 172.21.15.120:0/4172900607 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:03.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.172+0000 7f91ee7fc640 10 monclient(hunting): get_auth_request con 0x7f91f013f960 auth_method 0 2024-02-06T22:18:03.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.172+0000 7f91ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:03.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.172+0000 7f91ee7fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:03.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.172+0000 7f91ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:03.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.172+0000 7f91eeffd640 10 monclient(hunting): get_auth_request con 0x7f91f00b5230 auth_method 0 2024-02-06T22:18:03.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.172+0000 7f91eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:03.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.172+0000 7f91eeffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:03.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.172+0000 7f91eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:03.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.173+0000 7f91ee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:03.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.173+0000 7f91ee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:03.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.173+0000 7f91ee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:03.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.173+0000 7f91eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:03.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.173+0000 7f91eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:03.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.173+0000 7f91eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:03.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.175+0000 7f91ee7fc640 10 monclient(hunting): handle_auth_done global_id 21058 payload 824 2024-02-06T22:18:03.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.175+0000 7f91ee7fc640 1 -- 172.21.15.120:0/4172900607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91f00b5230 msgr2=0x7f91f0133300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:03.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.175+0000 7f91ee7fc640 1 --2- 172.21.15.120:0/4172900607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91f00b5230 0x7f91f0133300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.175+0000 7f91ee7fc640 1 -- 172.21.15.120:0/4172900607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91f0142d00 msgr2=0x7f91f00acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:03.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.175+0000 7f91ee7fc640 1 --2- 172.21.15.120:0/4172900607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91f0142d00 0x7f91f00acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.175+0000 7f91ee7fc640 10 monclient: _finish_hunting 0 2024-02-06T22:18:03.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.175+0000 7f91ee7fc640 1 monclient: found mon.noname-b 2024-02-06T22:18:03.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.175+0000 7f91ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:03.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.175+0000 7f91ee7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T22:18:03.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.175+0000 7f91ee7fc640 1 -- 172.21.15.120:0/4172900607 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e00094d0 con 0x7f91f013f960 2024-02-06T22:18:03.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.177+0000 7f91ee7fc640 1 --2- 172.21.15.120:0/4172900607 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f013f960 0x7f91f0133840 secure :-1 s=READY pgs=5094 cs=0 l=1 rev1=1 crypto rx=0x7f91d800c410 tx=0x7f91d800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:03.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.177+0000 7f91e77fe640 10 client.?.objecter ms_handle_connect 0x7f91f013f960 2024-02-06T22:18:03.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.177+0000 7f91e77fe640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:03.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91e77fe640 1 -- 172.21.15.120:0/4172900607 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d80035f0 con 0x7f91f013f960 2024-02-06T22:18:03.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:03.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:18:03.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91e77fe640 10 monclient: dump: 2024-02-06T22:18:03.190 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:03.190 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:03.190 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.190 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.190 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:03.190 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:03.190 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:03.190 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:03.190 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:03.190 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:03.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91e77fe640 10 monclient: _finish_auth 0 2024-02-06T22:18:03.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91e77fe640 10 monclient: _check_auth_tickets 2024-02-06T22:18:03.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:03.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91e77fe640 1 -- 172.21.15.120:0/4172900607 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d8003750 con 0x7f91f013f960 2024-02-06T22:18:03.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91e77fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:03.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91f5274640 5 monclient: authenticate success, global_id 21058 2024-02-06T22:18:03.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91e77fe640 1 -- 172.21.15.120:0/4172900607 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8003b40 con 0x7f91f013f960 2024-02-06T22:18:03.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91f5274640 10 monclient: _renew_subs 2024-02-06T22:18:03.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91f5274640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:03.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91f5274640 1 -- 172.21.15.120:0/4172900607 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f013b080 con 0x7f91f013f960 2024-02-06T22:18:03.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91ecff9640 4 set_mon_vals no callback set 2024-02-06T22:18:03.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:03.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T22:18:03.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91e77fe640 10 monclient: dump: 2024-02-06T22:18:03.191 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:03.191 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:03.191 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.191 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.192 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:03.192 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:03.192 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:03.192 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:03.192 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:03.192 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:03.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91f5274640 10 client.21058.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:03.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91f5274640 10 monclient: _renew_subs 2024-02-06T22:18:03.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91f5274640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:03.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91f5274640 1 -- 172.21.15.120:0/4172900607 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f013c340 con 0x7f91f013f960 2024-02-06T22:18:03.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.181+0000 7f91f5274640 1 librados: init done 2024-02-06T22:18:03.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.182+0000 7f91f5274640 10 client.21058.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:03.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.182+0000 7f91f5274640 15 client.21058.objecter _session_command_op_assign -1 1 2024-02-06T22:18:03.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.182+0000 7f91f5274640 20 client.21058.objecter _get_session osd=-1 returning homeless 2024-02-06T22:18:03.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.182+0000 7f91f5274640 10 client.21058.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:03.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.182+0000 7f91f5274640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:03.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.182+0000 7f91f5274640 1 -- 172.21.15.120:0/4172900607 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f91f014cea0 con 0x7f91f013f960 2024-02-06T22:18:03.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.199+0000 7f91e77fe640 1 -- 172.21.15.120:0/4172900607 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f91d8003ca0 con 0x7f91f013f960 2024-02-06T22:18:03.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.199+0000 7f91e77fe640 1 --2- 172.21.15.120:0/4172900607 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f91d006d1f0 0x7f91d006f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:03.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.200+0000 7f91eeffd640 1 --2- 172.21.15.120:0/4172900607 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f91d006d1f0 0x7f91d006f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:03.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.200+0000 7f91eeffd640 10 monclient: get_auth_request con 0x7f91d006d1f0 auth_method 0 2024-02-06T22:18:03.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.200+0000 7f91e77fe640 10 client.21058.objecter ms_handle_connect 0x7f91d006d1f0 2024-02-06T22:18:03.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.200+0000 7f91eeffd640 1 --2- 172.21.15.120:0/4172900607 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f91d006d1f0 0x7f91d006f6b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f91e800a400 tx=0x7f91e800d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:03.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 1 -- 172.21.15.120:0/4172900607 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f91d80010c0 con 0x7f91f013f960 2024-02-06T22:18:03.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 10 client.21058.objecter ms_dispatch 0x7f91f0000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:03.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 3 client.21058.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:03.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 3 client.21058.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:03.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 10 client.21058.objecter checking command 1 2024-02-06T22:18:03.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 1 --2- 172.21.15.120:0/4172900607 >> [v2:172.21.15.120:6800/3324566974,v1:172.21.15.120:6801/3324566974] conn(0x7f91d0074f80 0x7f91d0077420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:03.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 20 client.21058.objecter _get_session s=0x7f91d00748b0 osd=3 3 2024-02-06T22:18:03.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 20 client.21058.objecter put_session s=0x7f91d00748b0 osd=3 3 2024-02-06T22:18:03.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 15 client.21058.objecter _session_command_op_remove -1 1 2024-02-06T22:18:03.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 20 client.21058.objecter _get_session s=0x7f91d00748b0 osd=3 3 2024-02-06T22:18:03.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 20 client.21058.objecter get_session s=0x7f91d00748b0 osd=3 3 2024-02-06T22:18:03.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 15 client.21058.objecter _session_command_op_assign 3 1 2024-02-06T22:18:03.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 20 client.21058.objecter put_session s=0x7f91d00748b0 osd=3 4 2024-02-06T22:18:03.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 10 client.21058.objecter _send_command 1 2024-02-06T22:18:03.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 1 -- 172.21.15.120:0/4172900607 --> [v2:172.21.15.120:6800/3324566974,v1:172.21.15.120:6801/3324566974] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91d0077af0 con 0x7f91d0074f80 2024-02-06T22:18:03.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.202+0000 7f91e77fe640 20 client.21058.objecter dump_active .. 0 homeless 2024-02-06T22:18:03.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.203+0000 7f91ef7fe640 1 --2- 172.21.15.120:0/4172900607 >> [v2:172.21.15.120:6800/3324566974,v1:172.21.15.120:6801/3324566974] conn(0x7f91d0074f80 0x7f91d0077420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:03.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.203+0000 7f91ef7fe640 10 monclient: get_auth_request con 0x7f91d0074f80 auth_method 0 2024-02-06T22:18:03.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.206+0000 7f91e77fe640 10 client.21058.objecter ms_handle_connect 0x7f91d0074f80 2024-02-06T22:18:03.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.206+0000 7f91ef7fe640 1 --2- 172.21.15.120:0/4172900607 >> [v2:172.21.15.120:6800/3324566974,v1:172.21.15.120:6801/3324566974] conn(0x7f91d0074f80 0x7f91d0077420 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:03.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.230+0000 7f91e77fe640 1 -- 172.21.15.120:0/4172900607 <== osd.3 v2:172.21.15.120:6800/3324566974 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f91d0077af0 con 0x7f91d0074f80 2024-02-06T22:18:03.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.230+0000 7f91e77fe640 10 client.21058.objecter ms_dispatch 0x7f91f0000b90 command_reply(tid 1: 0 ) v1 2024-02-06T22:18:03.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.230+0000 7f91e77fe640 10 client.21058.objecter _finish_command 1 = system:0 2024-02-06T22:18:03.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.230+0000 7f91e77fe640 20 client.21058.objecter put_session s=0x7f91d00748b0 osd=3 4 2024-02-06T22:18:03.230 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.230+0000 7f91e77fe640 15 client.21058.objecter _session_command_op_remove 3 1 2024-02-06T22:18:03.245 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.245+0000 7f91f5274640 10 client.21058.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-02-06T22:18:03.245 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.245+0000 7f91f5274640 15 client.21058.objecter _session_command_op_assign -1 2 2024-02-06T22:18:03.245 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.245+0000 7f91f5274640 20 client.21058.objecter _get_session s=0x7f91d00748b0 osd=3 3 2024-02-06T22:18:03.245 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.245+0000 7f91f5274640 20 client.21058.objecter put_session s=0x7f91d00748b0 osd=3 3 2024-02-06T22:18:03.246 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.245+0000 7f91f5274640 20 client.21058.objecter _get_session s=0x7f91d00748b0 osd=3 3 2024-02-06T22:18:03.246 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.245+0000 7f91f5274640 15 client.21058.objecter _session_command_op_remove -1 2 2024-02-06T22:18:03.246 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.245+0000 7f91f5274640 20 client.21058.objecter get_session s=0x7f91d00748b0 osd=3 3 2024-02-06T22:18:03.246 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.245+0000 7f91f5274640 15 client.21058.objecter _session_command_op_assign 3 2 2024-02-06T22:18:03.246 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.245+0000 7f91f5274640 20 client.21058.objecter put_session s=0x7f91d00748b0 osd=3 4 2024-02-06T22:18:03.246 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.245+0000 7f91f5274640 10 client.21058.objecter _send_command 2 2024-02-06T22:18:03.246 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.245+0000 7f91f5274640 1 -- 172.21.15.120:0/4172900607 --> [v2:172.21.15.120:6800/3324566974,v1:172.21.15.120:6801/3324566974] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f91f00ac810 con 0x7f91d0074f80 2024-02-06T22:18:03.260 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.259+0000 7f91e77fe640 1 -- 172.21.15.120:0/4172900607 <== osd.3 v2:172.21.15.120:6800/3324566974 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f91f00ac810 con 0x7f91d0074f80 2024-02-06T22:18:03.260 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91e77fe640 10 client.21058.objecter ms_dispatch 0x7f91f0000b90 command_reply(tid 2: 0 ) v1 2024-02-06T22:18:03.260 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91e77fe640 10 client.21058.objecter _finish_command 2 = system:0 2024-02-06T22:18:03.260 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91e77fe640 20 client.21058.objecter put_session s=0x7f91d00748b0 osd=3 4 2024-02-06T22:18:03.260 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91e77fe640 15 client.21058.objecter _session_command_op_remove 3 2 2024-02-06T22:18:03.261 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-02-06T22:18:03.261 INFO:teuthology.orchestra.run.smithi120.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-06T22:18:03.261 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-02-06T22:18:03.262 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 10 librados: watch_flush enter 2024-02-06T22:18:03.262 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 10 librados: watch_flush exit 2024-02-06T22:18:03.262 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 10 client.21058.objecter close_session for osd.3 2024-02-06T22:18:03.263 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 1 -- 172.21.15.120:0/4172900607 >> [v2:172.21.15.120:6800/3324566974,v1:172.21.15.120:6801/3324566974] conn(0x7f91d0074f80 msgr2=0x7f91d0077420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:03.263 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 1 --2- 172.21.15.120:0/4172900607 >> [v2:172.21.15.120:6800/3324566974,v1:172.21.15.120:6801/3324566974] conn(0x7f91d0074f80 0x7f91d0077420 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.263 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 20 client.21058.objecter put_session s=0x7f91d00748b0 osd=3 1 2024-02-06T22:18:03.263 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 20 client.21058.objecter shutdown clearing up homeless session... 2024-02-06T22:18:03.263 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 10 client.21058.objecter successfully canceled tick 2024-02-06T22:18:03.263 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 1 -- 172.21.15.120:0/4172900607 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f91d006d1f0 msgr2=0x7f91d006f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:03.263 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 1 --2- 172.21.15.120:0/4172900607 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f91d006d1f0 0x7f91d006f6b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f91e800a400 tx=0x7f91e800d040 comp rx=0 tx=0).stop 2024-02-06T22:18:03.263 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 10 monclient: shutdown 2024-02-06T22:18:03.263 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 20 monclient: shutdown canceling and discarding version request 1 2024-02-06T22:18:03.263 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:03.263 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 1 -- 172.21.15.120:0/4172900607 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f013f960 msgr2=0x7f91f0133840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:03.263 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 1 --2- 172.21.15.120:0/4172900607 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f013f960 0x7f91f0133840 secure :-1 s=READY pgs=5094 cs=0 l=1 rev1=1 crypto rx=0x7f91d800c410 tx=0x7f91d800c8d0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.263 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91ef7fe640 1 -- 172.21.15.120:0/4172900607 reap_dead start 2024-02-06T22:18:03.264 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 1 -- 172.21.15.120:0/4172900607 shutdown_connections 2024-02-06T22:18:03.264 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 1 -- 172.21.15.120:0/4172900607 >> 172.21.15.120:0/4172900607 conn(0x7f91f006d1c0 msgr2=0x7f91f00b3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:03.264 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 1 -- 172.21.15.120:0/4172900607 shutdown_connections 2024-02-06T22:18:03.264 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 1 -- 172.21.15.120:0/4172900607 wait complete. 2024-02-06T22:18:03.264 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.260+0000 7f91f5274640 1 librados: shutdown 2024-02-06T22:18:03.273 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-02-06T22:18:03.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.375+0000 7f2a27d54640 10 monclient: get_monmap_and_config 2024-02-06T22:18:03.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.375+0000 7f2a27d54640 10 monclient: build_initial_monmap 2024-02-06T22:18:03.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.375+0000 7f2a27d54640 10 monclient: monmap: 2024-02-06T22:18:03.376 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:03.376 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:03.377 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:03.375534+0000 2024-02-06T22:18:03.377 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:03.375534+0000 2024-02-06T22:18:03.377 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:03.377 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:03.377 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:03.378 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:03.378 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:03.378 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:03.378 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:03.378 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:03.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.375+0000 7f2a27d54640 1 Processor -- start 2024-02-06T22:18:03.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 1 -- start start 2024-02-06T22:18:03.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 10 monclient: init 2024-02-06T22:18:03.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:03.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-02-06T22:18:03.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a200b54b0 addr v1:172.21.15.135:6789/0 2024-02-06T22:18:03.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2a2013ff40 0x7f2a200a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:03.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 10 monclient(hunting): picked mon.noname-a con 0x7f2a2013ff40 addr v2:172.21.15.120:3300/0 2024-02-06T22:18:03.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2a20142680 0x7f2a20142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:03.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 10 monclient(hunting): picked mon.noname-c con 0x7f2a20142680 addr v2:172.21.15.135:3300/0 2024-02-06T22:18:03.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a200ab720 con 0x7f2a200b54b0 2024-02-06T22:18:03.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:03.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a200ab520 con 0x7f2a2013ff40 2024-02-06T22:18:03.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:03.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a200b5890 con 0x7f2a20142680 2024-02-06T22:18:03.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:03.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.376+0000 7f2a27d54640 10 monclient(hunting): authenticate will time out at 3513.713189s 2024-02-06T22:18:03.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.377+0000 7f2a252ef640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2a2013ff40 0x7f2a200a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:03.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.377+0000 7f2a262f1640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2a20142680 0x7f2a20142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:03.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.377+0000 7f2a252ef640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2a2013ff40 0x7f2a200a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:49576/0 (socket says 172.21.15.120:49576) 2024-02-06T22:18:03.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.377+0000 7f2a252ef640 1 -- 172.21.15.120:0/1371664952 learned_addr learned my addr 172.21.15.120:0/1371664952 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:03.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.377+0000 7f2a252ef640 10 monclient(hunting): get_auth_request con 0x7f2a2013ff40 auth_method 0 2024-02-06T22:18:03.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.377+0000 7f2a252ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:03.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.377+0000 7f2a252ef640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:03.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.377+0000 7f2a252ef640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:03.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.377+0000 7f2a262f1640 10 monclient(hunting): get_auth_request con 0x7f2a20142680 auth_method 0 2024-02-06T22:18:03.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.377+0000 7f2a262f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:03.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.377+0000 7f2a262f1640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:03.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.377+0000 7f2a262f1640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:03.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.378+0000 7f2a252ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:03.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.378+0000 7f2a252ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:03.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.378+0000 7f2a252ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:03.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.378+0000 7f2a262f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:03.385 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.378+0000 7f2a262f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:03.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.378+0000 7f2a262f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:03.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.379+0000 7f2a252ef640 10 monclient(hunting): handle_auth_done global_id 16542 payload 274 2024-02-06T22:18:03.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.379+0000 7f2a252ef640 1 -- 172.21.15.120:0/1371664952 >> v2:172.21.15.135:3300/0 conn(0x7f2a20142680 msgr2=0x7f2a20142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:03.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.379+0000 7f2a252ef640 1 --2- 172.21.15.120:0/1371664952 >> v2:172.21.15.135:3300/0 conn(0x7f2a20142680 0x7f2a20142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.379+0000 7f2a252ef640 1 -- 172.21.15.120:0/1371664952 >> v1:172.21.15.135:6789/0 conn(0x7f2a200b54b0 legacy=0x7f2a2013f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:03.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.379+0000 7f2a252ef640 10 monclient: _finish_hunting 0 2024-02-06T22:18:03.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.379+0000 7f2a252ef640 1 monclient: found mon.noname-a 2024-02-06T22:18:03.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.379+0000 7f2a252ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:03.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.379+0000 7f2a252ef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:18:03.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.379+0000 7f2a252ef640 1 -- 172.21.15.120:0/1371664952 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a200b4800 con 0x7f2a2013ff40 2024-02-06T22:18:03.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.382+0000 7f2a252ef640 1 --2- 172.21.15.120:0/1371664952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a2013ff40 0x7f2a200a9860 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7f2a200aa6a0 tx=0x7f2a1002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d29a2557c1c75d8 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:03.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a24aee640 1 -- 172.21.15.120:0/1371664952 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a10030860 con 0x7f2a2013ff40 2024-02-06T22:18:03.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a24aee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:03.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a24aee640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:18:03.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a24aee640 10 monclient: dump: 2024-02-06T22:18:03.389 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:03.389 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:03.389 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.389 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.389 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:03.389 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:03.390 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:03.390 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:03.390 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:03.390 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:03.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a24aee640 10 monclient: _finish_auth 0 2024-02-06T22:18:03.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a24aee640 10 monclient: _check_auth_tickets 2024-02-06T22:18:03.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a24aee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:03.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a24aee640 1 -- 172.21.15.120:0/1371664952 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a100309c0 con 0x7f2a2013ff40 2024-02-06T22:18:03.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a24aee640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:03.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a24aee640 1 -- 172.21.15.120:0/1371664952 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a10030d50 con 0x7f2a2013ff40 2024-02-06T22:18:03.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a27d54640 5 monclient: authenticate success, global_id 16542 2024-02-06T22:18:03.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a27d54640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:03.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a27d54640 4 set_mon_vals no callback set 2024-02-06T22:18:03.390 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a24aee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:03.391 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a24aee640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:18:03.391 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a24aee640 10 monclient: dump: 2024-02-06T22:18:03.391 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:03.391 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:03.391 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.391 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.391 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:03.391 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:03.391 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:03.391 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:03.391 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:03.391 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:03.391 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a27d54640 10 monclient: shutdown 2024-02-06T22:18:03.391 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a27d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:03.391 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a27d54640 1 -- 172.21.15.120:0/1371664952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a2013ff40 msgr2=0x7f2a200a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:03.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a27d54640 1 --2- 172.21.15.120:0/1371664952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a2013ff40 0x7f2a200a9860 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7f2a200aa6a0 tx=0x7f2a1002b9a0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a27d54640 1 -- 172.21.15.120:0/1371664952 shutdown_connections 2024-02-06T22:18:03.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a27d54640 1 --2- 172.21.15.120:0/1371664952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a2013ff40 0x7f2a200a9860 unknown :-1 s=CLOSED pgs=3351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a27d54640 1 --2- 172.21.15.120:0/1371664952 >> v2:172.21.15.135:3300/0 conn(0x7f2a20142680 0x7f2a20142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.386+0000 7f2a27d54640 1 -- 172.21.15.120:0/1371664952 >> 172.21.15.120:0/1371664952 conn(0x7f2a200b8660 msgr2=0x7f2a200b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:03.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.387+0000 7f2a27d54640 1 -- 172.21.15.120:0/1371664952 shutdown_connections 2024-02-06T22:18:03.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.387+0000 7f2a27d54640 1 -- 172.21.15.120:0/1371664952 wait complete. 2024-02-06T22:18:03.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.387+0000 7f2a27d54640 10 monclient: build_initial_monmap 2024-02-06T22:18:03.392 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.387+0000 7f2a27d54640 10 monclient: monmap: 2024-02-06T22:18:03.392 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:03.393 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:03.393 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:03.393 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:03.393 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:03.393 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:03.393 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:03.393 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:03.393 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:03.393 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:03.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.388+0000 7f2a27d54640 1 librados: starting msgr at 2024-02-06T22:18:03.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.388+0000 7f2a27d54640 1 librados: starting objecter 2024-02-06T22:18:03.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.388+0000 7f2a27d54640 1 Processor -- start 2024-02-06T22:18:03.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.388+0000 7f2a27d54640 1 -- start start 2024-02-06T22:18:03.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.388+0000 7f2a27d54640 1 librados: setting wanted keys 2024-02-06T22:18:03.393 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.388+0000 7f2a27d54640 1 librados: calling monclient init 2024-02-06T22:18:03.394 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.388+0000 7f2a27d54640 10 monclient: init 2024-02-06T22:18:03.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:03.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T22:18:03.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a200b54b0 0x7f2a201331c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:03.396 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 10 monclient(hunting): picked mon.noname-b con 0x7f2a200b54b0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a2013ff40 0x7f2a201320f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 10 monclient(hunting): picked mon.noname-c con 0x7f2a2013ff40 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a20142680 0x7f2a20132630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 10 monclient(hunting): picked mon.noname-a con 0x7f2a20142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20132b70 con 0x7f2a20142680 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20134bd0 con 0x7f2a200b54b0 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20134eb0 con 0x7f2a2013ff40 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.389+0000 7f2a27d54640 10 monclient(hunting): authenticate will time out at 3513.726159s 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a252ef640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a2013ff40 0x7f2a201320f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a262f1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a20142680 0x7f2a20132630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a25af0640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a200b54b0 0x7f2a201331c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:03.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a25af0640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a200b54b0 0x7f2a201331c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:51132/0 (socket says 172.21.15.120:51132) 2024-02-06T22:18:03.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a25af0640 1 -- 172.21.15.120:0/3051832554 learned_addr learned my addr 172.21.15.120:0/3051832554 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:03.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a25af0640 10 monclient(hunting): get_auth_request con 0x7f2a200b54b0 auth_method 0 2024-02-06T22:18:03.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a25af0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:03.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a25af0640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:03.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a25af0640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:03.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a252ef640 10 monclient(hunting): get_auth_request con 0x7f2a2013ff40 auth_method 0 2024-02-06T22:18:03.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a252ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:03.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a252ef640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:03.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a252ef640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:03.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a262f1640 10 monclient(hunting): get_auth_request con 0x7f2a20142680 auth_method 0 2024-02-06T22:18:03.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a262f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:03.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a262f1640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:03.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a262f1640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:03.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a25af0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:03.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a25af0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:03.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.390+0000 7f2a25af0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:03.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.391+0000 7f2a262f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:03.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.391+0000 7f2a262f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:03.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.391+0000 7f2a262f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:03.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.391+0000 7f2a252ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:03.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.391+0000 7f2a252ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:03.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.391+0000 7f2a252ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:03.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.393+0000 7f2a25af0640 10 monclient(hunting): handle_auth_done global_id 21064 payload 824 2024-02-06T22:18:03.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.393+0000 7f2a25af0640 1 -- 172.21.15.120:0/3051832554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a2013ff40 msgr2=0x7f2a201320f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:03.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.393+0000 7f2a25af0640 1 --2- 172.21.15.120:0/3051832554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a2013ff40 0x7f2a201320f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.393+0000 7f2a25af0640 1 -- 172.21.15.120:0/3051832554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a20142680 msgr2=0x7f2a20132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:03.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.393+0000 7f2a25af0640 1 --2- 172.21.15.120:0/3051832554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a20142680 0x7f2a20132630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.393+0000 7f2a25af0640 10 monclient: _finish_hunting 0 2024-02-06T22:18:03.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.393+0000 7f2a25af0640 1 monclient: found mon.noname-b 2024-02-06T22:18:03.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.393+0000 7f2a25af0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:03.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.393+0000 7f2a25af0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T22:18:03.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.393+0000 7f2a25af0640 1 -- 172.21.15.120:0/3051832554 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a100094d0 con 0x7f2a200b54b0 2024-02-06T22:18:03.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.395+0000 7f2a25af0640 1 --2- 172.21.15.120:0/3051832554 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a200b54b0 0x7f2a201331c0 secure :-1 s=READY pgs=5096 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00c400 tx=0x7f2a1c00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:03.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.395+0000 7f2a167fc640 10 client.?.objecter ms_handle_connect 0x7f2a200b54b0 2024-02-06T22:18:03.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.396+0000 7f2a167fc640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:03.401 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a167fc640 1 -- 172.21.15.120:0/3051832554 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c0052d0 con 0x7f2a200b54b0 2024-02-06T22:18:03.401 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:03.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:18:03.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a167fc640 10 monclient: dump: 2024-02-06T22:18:03.402 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:03.402 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:03.402 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.402 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.402 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:03.402 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:03.402 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:03.402 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:03.402 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:03.402 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:03.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a167fc640 10 monclient: _finish_auth 0 2024-02-06T22:18:03.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a167fc640 10 monclient: _check_auth_tickets 2024-02-06T22:18:03.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:03.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a27d54640 5 monclient: authenticate success, global_id 21064 2024-02-06T22:18:03.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a167fc640 1 -- 172.21.15.120:0/3051832554 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1c005450 con 0x7f2a200b54b0 2024-02-06T22:18:03.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a167fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:03.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a167fc640 1 -- 172.21.15.120:0/3051832554 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c007260 con 0x7f2a200b54b0 2024-02-06T22:18:03.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a27d54640 10 monclient: _renew_subs 2024-02-06T22:18:03.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a27d54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:03.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a17fff640 4 set_mon_vals no callback set 2024-02-06T22:18:03.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a27d54640 1 -- 172.21.15.120:0/3051832554 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a20135190 con 0x7f2a200b54b0 2024-02-06T22:18:03.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:03.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T22:18:03.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a167fc640 10 monclient: dump: 2024-02-06T22:18:03.403 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:03.403 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:03.404 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.404 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:03.404 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:03.404 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:03.404 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:03.404 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:03.404 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:03.404 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:03.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a27d54640 10 client.21064.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:03.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a27d54640 10 monclient: _renew_subs 2024-02-06T22:18:03.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a27d54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:03.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a27d54640 1 -- 172.21.15.120:0/3051832554 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a20135d50 con 0x7f2a200b54b0 2024-02-06T22:18:03.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.399+0000 7f2a27d54640 1 librados: init done 2024-02-06T22:18:03.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.400+0000 7f2a27d54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:03.404 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.400+0000 7f2a27d54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:03.405 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.400+0000 7f2a27d54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:03.405 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.400+0000 7f2a27d54640 1 -- 172.21.15.120:0/3051832554 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2004ef90 con 0x7f2a200b54b0 2024-02-06T22:18:03.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.419+0000 7f2a167fc640 1 -- 172.21.15.120:0/3051832554 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2a1c0073c0 con 0x7f2a200b54b0 2024-02-06T22:18:03.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.420+0000 7f2a167fc640 1 --2- 172.21.15.120:0/3051832554 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f2a0406d1f0 0x7f2a0406f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:03.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.420+0000 7f2a252ef640 1 --2- 172.21.15.120:0/3051832554 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f2a0406d1f0 0x7f2a0406f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:03.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.420+0000 7f2a252ef640 10 monclient: get_auth_request con 0x7f2a0406d1f0 auth_method 0 2024-02-06T22:18:03.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.420+0000 7f2a167fc640 10 client.21064.objecter ms_handle_connect 0x7f2a0406d1f0 2024-02-06T22:18:03.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.420+0000 7f2a252ef640 1 --2- 172.21.15.120:0/3051832554 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f2a0406d1f0 0x7f2a0406f6b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2a10036000 tx=0x7f2a10008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:03.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.423+0000 7f2a167fc640 1 -- 172.21.15.120:0/3051832554 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f2a1c007670 con 0x7f2a200b54b0 2024-02-06T22:18:03.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.423+0000 7f2a167fc640 10 client.21064.objecter ms_dispatch 0x7f2a20000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:03.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.423+0000 7f2a167fc640 3 client.21064.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:03.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.423+0000 7f2a167fc640 3 client.21064.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:03.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.423+0000 7f2a167fc640 20 client.21064.objecter dump_active .. 0 homeless 2024-02-06T22:18:03.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.660+0000 7f2a167fc640 1 -- 172.21.15.120:0/3051832554 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2a1c0548d0 con 0x7f2a200b54b0 2024-02-06T22:18:03.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.660+0000 7f2a167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:03.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.660+0000 7f2a167fc640 10 monclient: _finish_command 1 = system:0 2024-02-06T22:18:03.804 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:43.618 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:03.807 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:43.622 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:03.833 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:43.648 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:03.844 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.843+0000 7f2a27d54640 1 -- 172.21.15.120:0/3051832554 --> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a200aa450 con 0x7f2a0406d1f0 2024-02-06T22:18:03.844 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.844+0000 7f2a167fc640 1 -- 172.21.15.120:0/3051832554 <== mgr.4100 v2:172.21.15.135:6800/315664144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2a10030660 con 0x7f2a0406d1f0 2024-02-06T22:18:03.844 INFO:teuthology.orchestra.run.smithi120.stderr:instructed osd(s) 3 to deep-scrub 2024-02-06T22:18:03.847 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 10 librados: watch_flush enter 2024-02-06T22:18:03.847 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 10 librados: watch_flush exit 2024-02-06T22:18:03.847 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 20 client.21064.objecter shutdown clearing up homeless session... 2024-02-06T22:18:03.847 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 10 client.21064.objecter successfully canceled tick 2024-02-06T22:18:03.848 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 1 -- 172.21.15.120:0/3051832554 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f2a0406d1f0 msgr2=0x7f2a0406f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:03.848 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 1 --2- 172.21.15.120:0/3051832554 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f2a0406d1f0 0x7f2a0406f6b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2a10036000 tx=0x7f2a10008030 comp rx=0 tx=0).stop 2024-02-06T22:18:03.848 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 10 monclient: shutdown 2024-02-06T22:18:03.848 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:03.848 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 1 -- 172.21.15.120:0/3051832554 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a200b54b0 msgr2=0x7f2a201331c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:03.848 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 1 --2- 172.21.15.120:0/3051832554 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a200b54b0 0x7f2a201331c0 secure :-1 s=READY pgs=5096 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00c400 tx=0x7f2a1c00c8c0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.848 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 1 -- 172.21.15.120:0/3051832554 shutdown_connections 2024-02-06T22:18:03.848 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 1 --2- 172.21.15.120:0/3051832554 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f2a0406d1f0 0x7f2a0406f6b0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.848 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 1 --2- 172.21.15.120:0/3051832554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a20142680 0x7f2a20132630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.848 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 1 --2- 172.21.15.120:0/3051832554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a2013ff40 0x7f2a201320f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.848 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 1 --2- 172.21.15.120:0/3051832554 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a200b54b0 0x7f2a201331c0 unknown :-1 s=CLOSED pgs=5096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:03.848 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 1 -- 172.21.15.120:0/3051832554 >> 172.21.15.120:0/3051832554 conn(0x7f2a200b8660 msgr2=0x7f2a2013f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:03.848 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 1 -- 172.21.15.120:0/3051832554 shutdown_connections 2024-02-06T22:18:03.848 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 1 -- 172.21.15.120:0/3051832554 wait complete. 2024-02-06T22:18:03.848 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.846+0000 7f2a27d54640 1 librados: shutdown 2024-02-06T22:18:03.853 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:43.617 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:03.856 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:43.620 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:03.859 INFO:tasks.ceph:Scrubbing osd.4 2024-02-06T22:18:03.859 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-02-06T22:18:03.863 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:43.678 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:03.867 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:43.682 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:03.882 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:43.646 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:03.900 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:43.715 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:03.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.967+0000 7f1605864640 10 monclient: get_monmap_and_config 2024-02-06T22:18:03.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.967+0000 7f1605864640 10 monclient: build_initial_monmap 2024-02-06T22:18:03.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.967+0000 7f1605864640 10 monclient: monmap: 2024-02-06T22:18:03.968 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:03.968 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:03.968 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:03.967849+0000 2024-02-06T22:18:03.968 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:03.967849+0000 2024-02-06T22:18:03.968 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:03.968 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:03.968 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:03.969 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:03.969 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:03.969 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:03.969 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:03.969 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:03.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.968+0000 7f1605864640 1 Processor -- start 2024-02-06T22:18:03.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.968+0000 7f1605864640 1 -- start start 2024-02-06T22:18:03.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.968+0000 7f1605864640 10 monclient: init 2024-02-06T22:18:03.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.968+0000 7f1605864640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:03.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.968+0000 7f1605864640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-02-06T22:18:03.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.968+0000 7f1605864640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16000b5230 addr v1:172.21.15.135:6789/0 2024-02-06T22:18:03.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.969+0000 7f1605864640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f160013f960 0x7f160013fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:03.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.969+0000 7f1605864640 10 monclient(hunting): picked mon.noname-a con 0x7f160013f960 addr v2:172.21.15.120:3300/0 2024-02-06T22:18:03.970 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.969+0000 7f1605864640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f16001425f0 0x7f1600142a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:03.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.969+0000 7f1605864640 10 monclient(hunting): picked mon.noname-c con 0x7f16001425f0 addr v2:172.21.15.135:3300/0 2024-02-06T22:18:03.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.969+0000 7f1605864640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16000ab3e0 con 0x7f16000b5230 2024-02-06T22:18:03.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.969+0000 7f1605864640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:03.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.969+0000 7f1605864640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16000b5610 con 0x7f160013f960 2024-02-06T22:18:03.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.969+0000 7f1605864640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:03.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.969+0000 7f1605864640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16000ab6c0 con 0x7f16001425f0 2024-02-06T22:18:03.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.969+0000 7f1605864640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:03.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.969+0000 7f1605864640 10 monclient(hunting): authenticate will time out at 3514.305650s 2024-02-06T22:18:03.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.994+0000 7f15f7fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f160013f960 0x7f160013fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:03.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.996+0000 7f15f7fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f160013f960 0x7f160013fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:49600/0 (socket says 172.21.15.120:49600) 2024-02-06T22:18:03.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.996+0000 7f15f7fff640 1 -- 172.21.15.120:0/2981448405 learned_addr learned my addr 172.21.15.120:0/2981448405 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:03.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.996+0000 7f15f7fff640 10 monclient(hunting): get_auth_request con 0x7f160013f960 auth_method 0 2024-02-06T22:18:03.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.996+0000 7f15f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:03.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.996+0000 7f15f7fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:03.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.996+0000 7f15f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:03.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.997+0000 7f15f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:03.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.997+0000 7f15f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:03.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.997+0000 7f15f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:03.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:03.998+0000 7f15ff7fe640 1 --2- 172.21.15.120:0/2981448405 >> v2:172.21.15.135:3300/0 conn(0x7f16001425f0 0x7f1600142a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:04.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.002+0000 7f15f7fff640 10 monclient(hunting): handle_auth_done global_id 16548 payload 274 2024-02-06T22:18:04.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.002+0000 7f15f7fff640 1 -- 172.21.15.120:0/2981448405 >> v2:172.21.15.135:3300/0 conn(0x7f16001425f0 msgr2=0x7f1600142a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:04.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.002+0000 7f15f7fff640 1 --2- 172.21.15.120:0/2981448405 >> v2:172.21.15.135:3300/0 conn(0x7f16001425f0 0x7f1600142a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.002+0000 7f15f7fff640 1 -- 172.21.15.120:0/2981448405 >> v1:172.21.15.135:6789/0 conn(0x7f16000b5230 legacy=0x7f16000a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:04.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.002+0000 7f15f7fff640 10 monclient: _finish_hunting 0 2024-02-06T22:18:04.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.002+0000 7f15f7fff640 1 monclient: found mon.noname-a 2024-02-06T22:18:04.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.002+0000 7f15f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:04.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.002+0000 7f15f7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:18:04.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.002+0000 7f15f7fff640 1 -- 172.21.15.120:0/2981448405 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16000aa040 con 0x7f160013f960 2024-02-06T22:18:04.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.019+0000 7f15f7fff640 1 --2- 172.21.15.120:0/2981448405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f160013f960 0x7f160013fdf0 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 crypto rx=0x7f1600143690 tx=0x7f15e802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5eb6f3d8ffd6812 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:04.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f15fe7fc640 1 -- 172.21.15.120:0/2981448405 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e8031750 con 0x7f160013f960 2024-02-06T22:18:04.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f15fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:04.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f15fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:18:04.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f15fe7fc640 10 monclient: dump: 2024-02-06T22:18:04.034 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:04.034 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:04.034 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.034 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.034 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:04.035 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:04.035 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:04.035 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:04.035 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:04.035 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:04.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f15fe7fc640 10 monclient: _finish_auth 0 2024-02-06T22:18:04.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f15fe7fc640 10 monclient: _check_auth_tickets 2024-02-06T22:18:04.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f15fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:04.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f15fe7fc640 1 -- 172.21.15.120:0/2981448405 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e80318b0 con 0x7f160013f960 2024-02-06T22:18:04.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f15fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:04.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f15fe7fc640 1 -- 172.21.15.120:0/2981448405 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e8031c40 con 0x7f160013f960 2024-02-06T22:18:04.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f1605864640 5 monclient: authenticate success, global_id 16548 2024-02-06T22:18:04.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f1605864640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:04.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f15fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:04.036 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f1605864640 4 set_mon_vals no callback set 2024-02-06T22:18:04.036 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f15fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:18:04.036 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.033+0000 7f15fe7fc640 10 monclient: dump: 2024-02-06T22:18:04.036 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:04.036 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:04.036 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.036 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.036 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:04.036 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:04.036 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:04.036 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:04.036 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:04.036 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:04.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.034+0000 7f1605864640 10 monclient: shutdown 2024-02-06T22:18:04.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.034+0000 7f1605864640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:04.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.034+0000 7f1605864640 1 -- 172.21.15.120:0/2981448405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f160013f960 msgr2=0x7f160013fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:04.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.034+0000 7f1605864640 1 --2- 172.21.15.120:0/2981448405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f160013f960 0x7f160013fdf0 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 crypto rx=0x7f1600143690 tx=0x7f15e802b7f0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.034+0000 7f1605864640 1 -- 172.21.15.120:0/2981448405 shutdown_connections 2024-02-06T22:18:04.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.034+0000 7f1605864640 1 --2- 172.21.15.120:0/2981448405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f160013f960 0x7f160013fdf0 unknown :-1 s=CLOSED pgs=3352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.034+0000 7f1605864640 1 --2- 172.21.15.120:0/2981448405 >> v2:172.21.15.135:3300/0 conn(0x7f16001425f0 0x7f1600142a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.034+0000 7f1605864640 1 -- 172.21.15.120:0/2981448405 >> 172.21.15.120:0/2981448405 conn(0x7f160006d1c0 msgr2=0x7f160006d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:04.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.034+0000 7f1605864640 1 -- 172.21.15.120:0/2981448405 shutdown_connections 2024-02-06T22:18:04.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.034+0000 7f1605864640 1 -- 172.21.15.120:0/2981448405 wait complete. 2024-02-06T22:18:04.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.035+0000 7f1605864640 10 monclient: build_initial_monmap 2024-02-06T22:18:04.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.035+0000 7f1605864640 10 monclient: monmap: 2024-02-06T22:18:04.040 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:04.040 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:04.040 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:04.040 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:04.040 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:04.040 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:04.040 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:04.040 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:04.040 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:04.040 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:04.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.036+0000 7f1605864640 1 librados: starting msgr at 2024-02-06T22:18:04.040 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.036+0000 7f1605864640 1 librados: starting objecter 2024-02-06T22:18:04.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.036+0000 7f1605864640 1 Processor -- start 2024-02-06T22:18:04.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.036+0000 7f1605864640 1 -- start start 2024-02-06T22:18:04.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.036+0000 7f1605864640 1 librados: setting wanted keys 2024-02-06T22:18:04.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.036+0000 7f1605864640 1 librados: calling monclient init 2024-02-06T22:18:04.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.036+0000 7f1605864640 10 monclient: init 2024-02-06T22:18:04.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:04.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T22:18:04.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16000b5230 0x7f160013acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:04.041 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 10 monclient(hunting): picked mon.noname-c con 0x7f16000b5230 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:04.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f160013f960 0x7f160013b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:04.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 10 monclient(hunting): picked mon.noname-b con 0x7f160013f960 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:04.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16001425f0 0x7f1600133190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:04.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 10 monclient(hunting): picked mon.noname-a con 0x7f16001425f0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:04.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:04.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160013c0f0 con 0x7f16001425f0 2024-02-06T22:18:04.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:04.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16000ac110 con 0x7f160013f960 2024-02-06T22:18:04.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:04.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160013bbd0 con 0x7f16000b5230 2024-02-06T22:18:04.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:04.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.037+0000 7f1605864640 10 monclient(hunting): authenticate will time out at 3514.373802s 2024-02-06T22:18:04.043 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.042+0000 7f15ff7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16001425f0 0x7f1600133190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:04.044 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.044+0000 7f15ff7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16001425f0 0x7f1600133190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:49612/0 (socket says 172.21.15.120:49612) 2024-02-06T22:18:04.044 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.044+0000 7f15ff7fe640 1 -- 172.21.15.120:0/813156838 learned_addr learned my addr 172.21.15.120:0/813156838 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:04.044 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.044+0000 7f15ff7fe640 10 monclient(hunting): get_auth_request con 0x7f16001425f0 auth_method 0 2024-02-06T22:18:04.044 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.044+0000 7f15ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:04.044 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.044+0000 7f15ff7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:04.044 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.044+0000 7f15ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:04.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.045+0000 7f15ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:04.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.045+0000 7f15ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:04.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.045+0000 7f15ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:04.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.049+0000 7f15ff7fe640 10 monclient(hunting): handle_auth_done global_id 16551 payload 824 2024-02-06T22:18:04.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.049+0000 7f15ff7fe640 1 -- 172.21.15.120:0/813156838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16000b5230 msgr2=0x7f160013acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:04.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.049+0000 7f15ff7fe640 1 --2- 172.21.15.120:0/813156838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16000b5230 0x7f160013acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.049+0000 7f15ff7fe640 1 -- 172.21.15.120:0/813156838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f160013f960 msgr2=0x7f160013b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:04.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.049+0000 7f15ff7fe640 1 --2- 172.21.15.120:0/813156838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f160013f960 0x7f160013b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.049+0000 7f15ff7fe640 10 monclient: _finish_hunting 0 2024-02-06T22:18:04.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.049+0000 7f15ff7fe640 1 monclient: found mon.noname-a 2024-02-06T22:18:04.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.050+0000 7f15ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:04.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.050+0000 7f15ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:18:04.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.050+0000 7f15ff7fe640 1 -- 172.21.15.120:0/813156838 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e80094d0 con 0x7f16001425f0 2024-02-06T22:18:04.056 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.056+0000 7f15ff7fe640 1 --2- 172.21.15.120:0/813156838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16001425f0 0x7f1600133190 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7f15f800cc30 tx=0x7f15f800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:04.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.056+0000 7f15fcff9640 10 client.?.objecter ms_handle_connect 0x7f16001425f0 2024-02-06T22:18:04.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.056+0000 7f15fcff9640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:04.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f15fcff9640 1 -- 172.21.15.120:0/813156838 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f800ce40 con 0x7f16001425f0 2024-02-06T22:18:04.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f15fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:04.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f15fcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:18:04.085 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f15fcff9640 10 monclient: dump: 2024-02-06T22:18:04.086 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:04.086 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:04.086 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.086 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.086 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:04.086 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:04.086 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:04.086 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:04.086 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:04.087 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:04.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f15fcff9640 10 monclient: _finish_auth 0 2024-02-06T22:18:04.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f15fcff9640 10 monclient: _check_auth_tickets 2024-02-06T22:18:04.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f15fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:04.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f1605864640 5 monclient: authenticate success, global_id 16551 2024-02-06T22:18:04.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f1605864640 10 monclient: _renew_subs 2024-02-06T22:18:04.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f1605864640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:04.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f1605864640 1 -- 172.21.15.120:0/813156838 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1600138250 con 0x7f16001425f0 2024-02-06T22:18:04.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f1605864640 10 client.16551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:04.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f1605864640 10 monclient: _renew_subs 2024-02-06T22:18:04.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f1605864640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:04.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f1605864640 1 -- 172.21.15.120:0/813156838 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16001338b0 con 0x7f16001425f0 2024-02-06T22:18:04.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.082+0000 7f1605864640 1 librados: init done 2024-02-06T22:18:04.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.083+0000 7f1605864640 10 client.16551.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:04.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.083+0000 7f1605864640 15 client.16551.objecter _session_command_op_assign -1 1 2024-02-06T22:18:04.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.083+0000 7f1605864640 20 client.16551.objecter _get_session osd=-1 returning homeless 2024-02-06T22:18:04.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.083+0000 7f1605864640 10 client.16551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:04.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.083+0000 7f1605864640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:04.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.083+0000 7f1605864640 1 -- 172.21.15.120:0/813156838 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f16000aae90 con 0x7f16001425f0 2024-02-06T22:18:04.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.084+0000 7f15fcff9640 1 -- 172.21.15.120:0/813156838 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16000aae90 con 0x7f16001425f0 2024-02-06T22:18:04.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.084+0000 7f15fcff9640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:04.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.084+0000 7f15fcff9640 1 -- 172.21.15.120:0/813156838 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8003b90 con 0x7f16001425f0 2024-02-06T22:18:04.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.084+0000 7f15fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:04.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.084+0000 7f15fcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:18:04.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.084+0000 7f15fcff9640 10 monclient: dump: 2024-02-06T22:18:04.088 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:04.088 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:04.088 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.088 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.089 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:04.089 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:04.089 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:04.089 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:04.089 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:04.089 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:04.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.084+0000 7f15fdffb640 4 set_mon_vals no callback set 2024-02-06T22:18:04.120 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.120+0000 7f15fcff9640 1 -- 172.21.15.120:0/813156838 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f15f8003cf0 con 0x7f16001425f0 2024-02-06T22:18:04.121 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.121+0000 7f15fcff9640 1 --2- 172.21.15.120:0/813156838 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f15e006d1a0 0x7f15e006f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:04.121 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.121+0000 7f15feffd640 1 --2- 172.21.15.120:0/813156838 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f15e006d1a0 0x7f15e006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:04.121 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.121+0000 7f15feffd640 10 monclient: get_auth_request con 0x7f15e006d1a0 auth_method 0 2024-02-06T22:18:04.122 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.121+0000 7f15feffd640 1 --2- 172.21.15.120:0/813156838 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f15e006d1a0 0x7f15e006f660 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f15f0001560 tx=0x7f15f0007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:04.122 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.121+0000 7f15fcff9640 10 client.16551.objecter ms_handle_connect 0x7f15e006d1a0 2024-02-06T22:18:04.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.134+0000 7f15fcff9640 1 -- 172.21.15.120:0/813156838 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f15f8059860 con 0x7f16001425f0 2024-02-06T22:18:04.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.134+0000 7f15fcff9640 10 client.16551.objecter ms_dispatch 0x7f1600000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:04.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.134+0000 7f15fcff9640 3 client.16551.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:04.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.134+0000 7f15fcff9640 3 client.16551.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:04.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.134+0000 7f15fcff9640 10 client.16551.objecter checking command 1 2024-02-06T22:18:04.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.135+0000 7f15fcff9640 1 --2- 172.21.15.120:0/813156838 >> [v2:172.21.15.135:6802/2191547918,v1:172.21.15.135:6803/2191547918] conn(0x7f15e0074f30 0x7f15e00773d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:04.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.135+0000 7f15fcff9640 20 client.16551.objecter _get_session s=0x7f15e0074860 osd=4 3 2024-02-06T22:18:04.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.135+0000 7f15fcff9640 20 client.16551.objecter put_session s=0x7f15e0074860 osd=4 3 2024-02-06T22:18:04.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.135+0000 7f15fcff9640 15 client.16551.objecter _session_command_op_remove -1 1 2024-02-06T22:18:04.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.135+0000 7f15fcff9640 20 client.16551.objecter _get_session s=0x7f15e0074860 osd=4 3 2024-02-06T22:18:04.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.135+0000 7f15fcff9640 20 client.16551.objecter get_session s=0x7f15e0074860 osd=4 3 2024-02-06T22:18:04.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.135+0000 7f15fcff9640 15 client.16551.objecter _session_command_op_assign 4 1 2024-02-06T22:18:04.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.135+0000 7f15fcff9640 20 client.16551.objecter put_session s=0x7f15e0074860 osd=4 4 2024-02-06T22:18:04.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.135+0000 7f15fcff9640 10 client.16551.objecter _send_command 1 2024-02-06T22:18:04.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.135+0000 7f15fcff9640 1 -- 172.21.15.120:0/813156838 --> [v2:172.21.15.135:6802/2191547918,v1:172.21.15.135:6803/2191547918] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15e0077aa0 con 0x7f15e0074f30 2024-02-06T22:18:04.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.135+0000 7f15fcff9640 20 client.16551.objecter dump_active .. 0 homeless 2024-02-06T22:18:04.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.135+0000 7f15fcff9640 1 -- 172.21.15.120:0/813156838 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_get_version_reply(handle=1 version=1346) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15f8055400 con 0x7f16001425f0 2024-02-06T22:18:04.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.135+0000 7f15fcff9640 10 monclient: handle_get_version_reply finishing 1 version 1346 2024-02-06T22:18:04.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.135+0000 7f15f7fff640 1 --2- 172.21.15.120:0/813156838 >> [v2:172.21.15.135:6802/2191547918,v1:172.21.15.135:6803/2191547918] conn(0x7f15e0074f30 0x7f15e00773d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:04.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.136+0000 7f15f7fff640 10 monclient: get_auth_request con 0x7f15e0074f30 auth_method 0 2024-02-06T22:18:04.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.138+0000 7f15f7fff640 1 --2- 172.21.15.120:0/813156838 >> [v2:172.21.15.135:6802/2191547918,v1:172.21.15.135:6803/2191547918] conn(0x7f15e0074f30 0x7f15e00773d0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:04.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.139+0000 7f15fcff9640 10 client.16551.objecter ms_handle_connect 0x7f15e0074f30 2024-02-06T22:18:04.163 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.163+0000 7f15fcff9640 1 -- 172.21.15.120:0/813156838 <== osd.4 v2:172.21.15.135:6802/2191547918 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f15e0077aa0 con 0x7f15e0074f30 2024-02-06T22:18:04.163 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.163+0000 7f15fcff9640 10 client.16551.objecter ms_dispatch 0x7f1600000b90 command_reply(tid 1: 0 ) v1 2024-02-06T22:18:04.163 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.163+0000 7f15fcff9640 10 client.16551.objecter _finish_command 1 = system:0 2024-02-06T22:18:04.163 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.163+0000 7f15fcff9640 20 client.16551.objecter put_session s=0x7f15e0074860 osd=4 4 2024-02-06T22:18:04.163 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.163+0000 7f15fcff9640 15 client.16551.objecter _session_command_op_remove 4 1 2024-02-06T22:18:04.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.181+0000 7f1605864640 10 client.16551.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-02-06T22:18:04.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.181+0000 7f1605864640 15 client.16551.objecter _session_command_op_assign -1 2 2024-02-06T22:18:04.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.181+0000 7f1605864640 20 client.16551.objecter _get_session s=0x7f15e0074860 osd=4 3 2024-02-06T22:18:04.182 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.181+0000 7f1605864640 20 client.16551.objecter put_session s=0x7f15e0074860 osd=4 3 2024-02-06T22:18:04.182 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.181+0000 7f1605864640 20 client.16551.objecter _get_session s=0x7f15e0074860 osd=4 3 2024-02-06T22:18:04.182 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.181+0000 7f1605864640 15 client.16551.objecter _session_command_op_remove -1 2 2024-02-06T22:18:04.182 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.181+0000 7f1605864640 20 client.16551.objecter get_session s=0x7f15e0074860 osd=4 3 2024-02-06T22:18:04.182 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.181+0000 7f1605864640 15 client.16551.objecter _session_command_op_assign 4 2 2024-02-06T22:18:04.182 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.181+0000 7f1605864640 20 client.16551.objecter put_session s=0x7f15e0074860 osd=4 4 2024-02-06T22:18:04.182 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.181+0000 7f1605864640 10 client.16551.objecter _send_command 2 2024-02-06T22:18:04.182 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.181+0000 7f1605864640 1 -- 172.21.15.120:0/813156838 --> [v2:172.21.15.135:6802/2191547918,v1:172.21.15.135:6803/2191547918] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f160014cea0 con 0x7f15e0074f30 2024-02-06T22:18:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.194+0000 7f15fcff9640 1 -- 172.21.15.120:0/813156838 <== osd.4 v2:172.21.15.135:6802/2191547918 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f15e8031a20 con 0x7f15e0074f30 2024-02-06T22:18:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.194+0000 7f15fcff9640 10 client.16551.objecter ms_dispatch 0x7f1600000b90 command_reply(tid 2: 0 ) v1 2024-02-06T22:18:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.194+0000 7f15fcff9640 10 client.16551.objecter _finish_command 2 = system:0 2024-02-06T22:18:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.194+0000 7f15fcff9640 20 client.16551.objecter put_session s=0x7f15e0074860 osd=4 4 2024-02-06T22:18:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.194+0000 7f15fcff9640 15 client.16551.objecter _session_command_op_remove 4 2 2024-02-06T22:18:04.195 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-02-06T22:18:04.195 INFO:teuthology.orchestra.run.smithi120.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-06T22:18:04.196 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-02-06T22:18:04.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 10 librados: watch_flush enter 2024-02-06T22:18:04.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 10 librados: watch_flush exit 2024-02-06T22:18:04.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 10 client.16551.objecter close_session for osd.4 2024-02-06T22:18:04.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 1 -- 172.21.15.120:0/813156838 >> [v2:172.21.15.135:6802/2191547918,v1:172.21.15.135:6803/2191547918] conn(0x7f15e0074f30 msgr2=0x7f15e00773d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:04.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 1 --2- 172.21.15.120:0/813156838 >> [v2:172.21.15.135:6802/2191547918,v1:172.21.15.135:6803/2191547918] conn(0x7f15e0074f30 0x7f15e00773d0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 20 client.16551.objecter put_session s=0x7f15e0074860 osd=4 1 2024-02-06T22:18:04.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 20 client.16551.objecter shutdown clearing up homeless session... 2024-02-06T22:18:04.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 10 client.16551.objecter successfully canceled tick 2024-02-06T22:18:04.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 1 -- 172.21.15.120:0/813156838 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f15e006d1a0 msgr2=0x7f15e006f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:04.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 1 --2- 172.21.15.120:0/813156838 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f15e006d1a0 0x7f15e006f660 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f15f0001560 tx=0x7f15f0007990 comp rx=0 tx=0).stop 2024-02-06T22:18:04.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 10 monclient: shutdown 2024-02-06T22:18:04.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:04.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 1 -- 172.21.15.120:0/813156838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16001425f0 msgr2=0x7f1600133190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:04.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 1 --2- 172.21.15.120:0/813156838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16001425f0 0x7f1600133190 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7f15f800cc30 tx=0x7f15f800a5c0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f15ff7fe640 1 -- 172.21.15.120:0/813156838 reap_dead start 2024-02-06T22:18:04.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 1 -- 172.21.15.120:0/813156838 shutdown_connections 2024-02-06T22:18:04.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 1 -- 172.21.15.120:0/813156838 >> 172.21.15.120:0/813156838 conn(0x7f160006d1c0 msgr2=0x7f16000b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:04.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 1 -- 172.21.15.120:0/813156838 shutdown_connections 2024-02-06T22:18:04.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 1 -- 172.21.15.120:0/813156838 wait complete. 2024-02-06T22:18:04.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.195+0000 7f1605864640 1 librados: shutdown 2024-02-06T22:18:04.210 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-02-06T22:18:04.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.357+0000 7f96b83ba640 10 monclient: get_monmap_and_config 2024-02-06T22:18:04.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.357+0000 7f96b83ba640 10 monclient: build_initial_monmap 2024-02-06T22:18:04.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.357+0000 7f96b83ba640 10 monclient: monmap: 2024-02-06T22:18:04.358 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:04.358 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:04.358 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:04.358200+0000 2024-02-06T22:18:04.359 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:04.358200+0000 2024-02-06T22:18:04.359 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:04.359 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:04.359 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:04.359 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:04.359 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:04.359 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:04.359 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:04.359 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:04.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.358+0000 7f96b83ba640 1 Processor -- start 2024-02-06T22:18:04.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.358+0000 7f96b83ba640 1 -- start start 2024-02-06T22:18:04.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.358+0000 7f96b83ba640 10 monclient: init 2024-02-06T22:18:04.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.359+0000 7f96b83ba640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:04.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.359+0000 7f96b83ba640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-02-06T22:18:04.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.359+0000 7f96b83ba640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f96b00b54b0 0x7f96b00a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:04.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.359+0000 7f96b83ba640 10 monclient(hunting): picked mon.noname-a con 0x7f96b00b54b0 addr v2:172.21.15.120:3300/0 2024-02-06T22:18:04.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.359+0000 7f96b83ba640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96b013f830 addr v1:172.21.15.120:6789/0 2024-02-06T22:18:04.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.359+0000 7f96b83ba640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96b0142680 0x7f96b0142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:04.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.359+0000 7f96b83ba640 10 monclient(hunting): picked mon.noname-b con 0x7f96b0142680 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:04.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.359+0000 7f96b83ba640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b00ab720 con 0x7f96b013f830 2024-02-06T22:18:04.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.359+0000 7f96b83ba640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:04.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.359+0000 7f96b83ba640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b00ab520 con 0x7f96b00b54b0 2024-02-06T22:18:04.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.359+0000 7f96b83ba640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:04.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.359+0000 7f96b83ba640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b00b5890 con 0x7f96b0142680 2024-02-06T22:18:04.362 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.359+0000 7f96b83ba640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:04.362 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.359+0000 7f96b83ba640 10 monclient(hunting): authenticate will time out at 3514.695986s 2024-02-06T22:18:04.366 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.366+0000 7f96b6957640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96b0142680 0x7f96b0142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:04.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.373+0000 7f96b6957640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96b0142680 0x7f96b0142ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:55802/0 (socket says 172.21.15.120:55802) 2024-02-06T22:18:04.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.373+0000 7f96b6957640 1 -- 172.21.15.120:0/2801173987 learned_addr learned my addr 172.21.15.120:0/2801173987 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:04.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.373+0000 7f96b6957640 10 monclient(hunting): get_auth_request con 0x7f96b0142680 auth_method 0 2024-02-06T22:18:04.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.373+0000 7f96b6957640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:04.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.373+0000 7f96b6957640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:04.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.373+0000 7f96b6957640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:04.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.375+0000 7f96b6957640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:04.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.375+0000 7f96b6957640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:04.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.375+0000 7f96b6957640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:04.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.381+0000 7f96b6957640 10 monclient(hunting): handle_auth_done global_id 18728 payload 274 2024-02-06T22:18:04.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.381+0000 7f96b6957640 1 -- 172.21.15.120:0/2801173987 >> v2:172.21.15.120:3300/0 conn(0x7f96b00b54b0 msgr2=0x7f96b00a9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:04.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.381+0000 7f96b6957640 1 --2- 172.21.15.120:0/2801173987 >> v2:172.21.15.120:3300/0 conn(0x7f96b00b54b0 0x7f96b00a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.381+0000 7f96b6957640 1 -- 172.21.15.120:0/2801173987 >> v1:172.21.15.120:6789/0 conn(0x7f96b013f830 legacy=0x7f96b013fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:04.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.381+0000 7f96b6957640 10 monclient: _finish_hunting 0 2024-02-06T22:18:04.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.381+0000 7f96b6957640 1 monclient: found mon.noname-b 2024-02-06T22:18:04.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.381+0000 7f96b6957640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:04.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.381+0000 7f96b6957640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T22:18:04.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.381+0000 7f96b6957640 1 -- 172.21.15.120:0/2801173987 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b00b4800 con 0x7f96b0142680 2024-02-06T22:18:04.391 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.390+0000 7f96b6957640 1 --2- 172.21.15.120:0/2801173987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96b0142680 0x7f96b0142ac0 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009e50 tx=0x7f96ac02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3060a530f0788816 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:04.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b5154640 1 -- 172.21.15.120:0/2801173987 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac02ca70 con 0x7f96b0142680 2024-02-06T22:18:04.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b5154640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:04.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b5154640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:18:04.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b5154640 10 monclient: dump: 2024-02-06T22:18:04.400 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:04.400 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:04.400 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.400 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.400 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:04.400 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:04.400 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:04.401 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:04.402 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:04.402 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:04.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b5154640 10 monclient: _finish_auth 0 2024-02-06T22:18:04.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b5154640 10 monclient: _check_auth_tickets 2024-02-06T22:18:04.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b5154640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:04.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b5154640 1 -- 172.21.15.120:0/2801173987 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ac02cbd0 con 0x7f96b0142680 2024-02-06T22:18:04.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b5154640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:04.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b5154640 1 -- 172.21.15.120:0/2801173987 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac0343f0 con 0x7f96b0142680 2024-02-06T22:18:04.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b5154640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:04.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b5154640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T22:18:04.402 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b5154640 10 monclient: dump: 2024-02-06T22:18:04.402 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:04.402 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:04.403 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.403 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.403 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:04.403 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:04.403 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:04.403 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:04.403 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:04.403 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:04.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b83ba640 5 monclient: authenticate success, global_id 18728 2024-02-06T22:18:04.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b83ba640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:04.403 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.399+0000 7f96b83ba640 4 set_mon_vals no callback set 2024-02-06T22:18:04.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.400+0000 7f96b83ba640 10 monclient: shutdown 2024-02-06T22:18:04.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.400+0000 7f96b83ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:04.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.400+0000 7f96b83ba640 1 -- 172.21.15.120:0/2801173987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96b0142680 msgr2=0x7f96b0142ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:04.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.400+0000 7f96b83ba640 1 --2- 172.21.15.120:0/2801173987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96b0142680 0x7f96b0142ac0 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009e50 tx=0x7f96ac02bed0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.400+0000 7f96b83ba640 1 -- 172.21.15.120:0/2801173987 shutdown_connections 2024-02-06T22:18:04.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.400+0000 7f96b83ba640 1 --2- 172.21.15.120:0/2801173987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96b0142680 0x7f96b0142ac0 unknown :-1 s=CLOSED pgs=3347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.400+0000 7f96b83ba640 1 --2- 172.21.15.120:0/2801173987 >> v2:172.21.15.120:3300/0 conn(0x7f96b00b54b0 0x7f96b00a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.400+0000 7f96b83ba640 1 -- 172.21.15.120:0/2801173987 >> 172.21.15.120:0/2801173987 conn(0x7f96b00b8660 msgr2=0x7f96b00b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:04.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.400+0000 7f96b83ba640 1 -- 172.21.15.120:0/2801173987 shutdown_connections 2024-02-06T22:18:04.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.400+0000 7f96b83ba640 1 -- 172.21.15.120:0/2801173987 wait complete. 2024-02-06T22:18:04.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.401+0000 7f96b83ba640 10 monclient: build_initial_monmap 2024-02-06T22:18:04.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.401+0000 7f96b83ba640 10 monclient: monmap: 2024-02-06T22:18:04.407 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:04.407 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:04.407 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:04.407 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:04.407 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:04.407 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:04.407 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:04.407 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:04.407 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:04.407 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:04.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.401+0000 7f96b83ba640 1 librados: starting msgr at 2024-02-06T22:18:04.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.401+0000 7f96b83ba640 1 librados: starting objecter 2024-02-06T22:18:04.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.401+0000 7f96b83ba640 1 Processor -- start 2024-02-06T22:18:04.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 1 -- start start 2024-02-06T22:18:04.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 1 librados: setting wanted keys 2024-02-06T22:18:04.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 1 librados: calling monclient init 2024-02-06T22:18:04.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 10 monclient: init 2024-02-06T22:18:04.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:04.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T22:18:04.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96b00b54b0 0x7f96b0138f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:04.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 10 monclient(hunting): picked mon.noname-b con 0x7f96b00b54b0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:04.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96b013f830 0x7f96b0135310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:04.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 10 monclient(hunting): picked mon.noname-c con 0x7f96b013f830 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:04.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96b0142680 0x7f96b0135850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:04.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 10 monclient(hunting): picked mon.noname-a con 0x7f96b0142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:04.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:04.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b0133070 con 0x7f96b0142680 2024-02-06T22:18:04.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:04.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b0135d90 con 0x7f96b00b54b0 2024-02-06T22:18:04.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:04.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b0134c10 con 0x7f96b013f830 2024-02-06T22:18:04.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:04.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.402+0000 7f96b83ba640 10 monclient(hunting): authenticate will time out at 3514.739466s 2024-02-06T22:18:04.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.404+0000 7f96b6156640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96b00b54b0 0x7f96b0138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:04.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.406+0000 7f96b5955640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96b013f830 0x7f96b0135310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:04.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.408+0000 7f96b6156640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96b00b54b0 0x7f96b0138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:51156/0 (socket says 172.21.15.120:51156) 2024-02-06T22:18:04.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.408+0000 7f96b6156640 1 -- 172.21.15.120:0/3660739897 learned_addr learned my addr 172.21.15.120:0/3660739897 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:04.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.408+0000 7f96b6156640 10 monclient(hunting): get_auth_request con 0x7f96b00b54b0 auth_method 0 2024-02-06T22:18:04.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.408+0000 7f96b6156640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:04.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.408+0000 7f96b6156640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:04.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.408+0000 7f96b6156640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:04.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.409+0000 7f96b5955640 10 monclient(hunting): get_auth_request con 0x7f96b013f830 auth_method 0 2024-02-06T22:18:04.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.409+0000 7f96b5955640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:04.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.409+0000 7f96b5955640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:04.410 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.409+0000 7f96b5955640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:04.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.411+0000 7f96b6156640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:04.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.411+0000 7f96b6156640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:04.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.411+0000 7f96b6156640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:04.411 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.411+0000 7f96b5955640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:04.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.411+0000 7f96b5955640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:04.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.411+0000 7f96b5955640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:04.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.414+0000 7f96b5955640 10 monclient(hunting): handle_auth_done global_id 18731 payload 824 2024-02-06T22:18:04.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.414+0000 7f96b5955640 1 -- 172.21.15.120:0/3660739897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96b00b54b0 msgr2=0x7f96b0138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:04.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.414+0000 7f96b5955640 1 --2- 172.21.15.120:0/3660739897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96b00b54b0 0x7f96b0138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.414+0000 7f96b5955640 1 -- 172.21.15.120:0/3660739897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96b0142680 msgr2=0x7f96b0135850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:04.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.414+0000 7f96b5955640 1 --2- 172.21.15.120:0/3660739897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96b0142680 0x7f96b0135850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.414+0000 7f96b5955640 10 monclient: _finish_hunting 0 2024-02-06T22:18:04.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.414+0000 7f96b5955640 1 monclient: found mon.noname-c 2024-02-06T22:18:04.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.414+0000 7f96b5955640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:04.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.414+0000 7f96b5955640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T22:18:04.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.414+0000 7f96b5955640 1 -- 172.21.15.120:0/3660739897 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac009b30 con 0x7f96b013f830 2024-02-06T22:18:04.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.419+0000 7f96b5955640 1 --2- 172.21.15.120:0/3660739897 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96b013f830 0x7f96b0135310 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7f96a000c410 tx=0x7f96a000c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:04.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.419+0000 7f96a6ffd640 10 client.?.objecter ms_handle_connect 0x7f96b013f830 2024-02-06T22:18:04.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.419+0000 7f96a6ffd640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:04.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.426+0000 7f96a6ffd640 1 -- 172.21.15.120:0/3660739897 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a00035f0 con 0x7f96b013f830 2024-02-06T22:18:04.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.426+0000 7f96a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:04.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.426+0000 7f96a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T22:18:04.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.426+0000 7f96a6ffd640 10 monclient: dump: 2024-02-06T22:18:04.428 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:04.428 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:04.428 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.428 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.428 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:04.428 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:04.428 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:04.428 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:04.428 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:04.428 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:04.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.426+0000 7f96a6ffd640 10 monclient: _finish_auth 0 2024-02-06T22:18:04.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.426+0000 7f96a6ffd640 10 monclient: _check_auth_tickets 2024-02-06T22:18:04.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.426+0000 7f96a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:04.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.426+0000 7f96b83ba640 5 monclient: authenticate success, global_id 18731 2024-02-06T22:18:04.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.426+0000 7f96b83ba640 10 monclient: _renew_subs 2024-02-06T22:18:04.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.426+0000 7f96b83ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:18:04.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.426+0000 7f96b83ba640 1 -- 172.21.15.120:0/3660739897 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b0134ef0 con 0x7f96b013f830 2024-02-06T22:18:04.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.427+0000 7f96b83ba640 10 client.18731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:04.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.427+0000 7f96b83ba640 10 monclient: _renew_subs 2024-02-06T22:18:04.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.427+0000 7f96b83ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:18:04.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.427+0000 7f96b83ba640 1 -- 172.21.15.120:0/3660739897 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b01337c0 con 0x7f96b013f830 2024-02-06T22:18:04.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.427+0000 7f96b83ba640 1 librados: init done 2024-02-06T22:18:04.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.427+0000 7f96b83ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:04.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.427+0000 7f96b83ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:04.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.427+0000 7f96b83ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:18:04.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.427+0000 7f96b83ba640 1 -- 172.21.15.120:0/3660739897 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b013a030 con 0x7f96b013f830 2024-02-06T22:18:04.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.429+0000 7f96a6ffd640 1 -- 172.21.15.120:0/3660739897 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a0007030 con 0x7f96b013f830 2024-02-06T22:18:04.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.429+0000 7f96a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:04.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.429+0000 7f96a6ffd640 1 -- 172.21.15.120:0/3660739897 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a0007400 con 0x7f96b013f830 2024-02-06T22:18:04.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.429+0000 7f96a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:04.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.429+0000 7f96a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T22:18:04.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.429+0000 7f96a6ffd640 10 monclient: dump: 2024-02-06T22:18:04.431 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:04.431 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:04.431 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.431 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:04.431 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:04.431 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:04.431 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:04.431 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:04.431 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:04.431 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:04.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.429+0000 7f96b4953640 4 set_mon_vals no callback set 2024-02-06T22:18:04.646 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:44.410 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:04.647 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:44.411 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:04.655 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:44.420 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:04.691 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.691+0000 7f96a6ffd640 1 -- 172.21.15.120:0/3660739897 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f96a0006790 con 0x7f96b013f830 2024-02-06T22:18:04.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.691+0000 7f96a6ffd640 1 --2- 172.21.15.120:0/3660739897 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f969406d1a0 0x7f969406f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:04.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.692+0000 7f96b6156640 1 --2- 172.21.15.120:0/3660739897 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f969406d1a0 0x7f969406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:04.692 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.692+0000 7f96b6156640 10 monclient: get_auth_request con 0x7f969406d1a0 auth_method 0 2024-02-06T22:18:04.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.692+0000 7f96a6ffd640 10 client.18731.objecter ms_handle_connect 0x7f969406d1a0 2024-02-06T22:18:04.693 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.692+0000 7f96b6156640 1 --2- 172.21.15.120:0/3660739897 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f969406d1a0 0x7f969406f660 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009a40 tx=0x7f96a80091c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:04.706 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.705+0000 7f96a6ffd640 1 -- 172.21.15.120:0/3660739897 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f96a00584c0 con 0x7f96b013f830 2024-02-06T22:18:04.706 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.705+0000 7f96a6ffd640 10 client.18731.objecter ms_dispatch 0x7f96b0000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:04.706 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.705+0000 7f96a6ffd640 3 client.18731.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:04.706 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.705+0000 7f96a6ffd640 3 client.18731.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:04.706 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.705+0000 7f96a6ffd640 20 client.18731.objecter dump_active .. 0 homeless 2024-02-06T22:18:04.706 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.705+0000 7f96a6ffd640 1 -- 172.21.15.120:0/3660739897 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f96a0054760 con 0x7f96b013f830 2024-02-06T22:18:04.706 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.705+0000 7f96a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:04.707 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.705+0000 7f96a6ffd640 10 monclient: _finish_command 1 = system:0 2024-02-06T22:18:04.802 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:44.616 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:04.803 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:44.618 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:04.820 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:44.635 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:04.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.933+0000 7f96b83ba640 1 -- 172.21.15.120:0/3660739897 --> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f96b00aa450 con 0x7f969406d1a0 2024-02-06T22:18:04.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.933+0000 7f96a6ffd640 1 -- 172.21.15.120:0/3660739897 <== mgr.4100 v2:172.21.15.135:6800/315664144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f96a800e910 con 0x7f969406d1a0 2024-02-06T22:18:04.934 INFO:teuthology.orchestra.run.smithi120.stderr:instructed osd(s) 4 to deep-scrub 2024-02-06T22:18:04.937 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 10 librados: watch_flush enter 2024-02-06T22:18:04.938 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 10 librados: watch_flush exit 2024-02-06T22:18:04.938 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 20 client.18731.objecter shutdown clearing up homeless session... 2024-02-06T22:18:04.938 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 10 client.18731.objecter successfully canceled tick 2024-02-06T22:18:04.938 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 1 -- 172.21.15.120:0/3660739897 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f969406d1a0 msgr2=0x7f969406f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:04.938 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 1 --2- 172.21.15.120:0/3660739897 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f969406d1a0 0x7f969406f660 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009a40 tx=0x7f96a80091c0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.938 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 10 monclient: shutdown 2024-02-06T22:18:04.938 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:04.938 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 1 -- 172.21.15.120:0/3660739897 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96b013f830 msgr2=0x7f96b0135310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:04.938 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 1 --2- 172.21.15.120:0/3660739897 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96b013f830 0x7f96b0135310 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7f96a000c410 tx=0x7f96a000c810 comp rx=0 tx=0).stop 2024-02-06T22:18:04.938 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 1 -- 172.21.15.120:0/3660739897 shutdown_connections 2024-02-06T22:18:04.938 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 1 --2- 172.21.15.120:0/3660739897 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f969406d1a0 0x7f969406f660 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.938 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 1 --2- 172.21.15.120:0/3660739897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96b0142680 0x7f96b0135850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.939 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 1 --2- 172.21.15.120:0/3660739897 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96b013f830 0x7f96b0135310 unknown :-1 s=CLOSED pgs=3348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.939 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 1 --2- 172.21.15.120:0/3660739897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96b00b54b0 0x7f96b0138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:04.939 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 1 -- 172.21.15.120:0/3660739897 >> 172.21.15.120:0/3660739897 conn(0x7f96b00b8660 msgr2=0x7f96b013c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:04.939 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 1 -- 172.21.15.120:0/3660739897 shutdown_connections 2024-02-06T22:18:04.939 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 1 -- 172.21.15.120:0/3660739897 wait complete. 2024-02-06T22:18:04.939 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:04.936+0000 7f96b83ba640 1 librados: shutdown 2024-02-06T22:18:04.949 INFO:tasks.ceph:Scrubbing osd.5 2024-02-06T22:18:04.949 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-02-06T22:18:05.058 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.058+0000 7fb18179d640 10 monclient: get_monmap_and_config 2024-02-06T22:18:05.058 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.058+0000 7fb18179d640 10 monclient: build_initial_monmap 2024-02-06T22:18:05.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.058+0000 7fb18179d640 10 monclient: monmap: 2024-02-06T22:18:05.059 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:05.059 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:05.059 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:05.058341+0000 2024-02-06T22:18:05.059 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:05.058341+0000 2024-02-06T22:18:05.059 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:05.059 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.059 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:05.059 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:05.059 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:05.059 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:05.059 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:05.059 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.058+0000 7fb18179d640 1 Processor -- start 2024-02-06T22:18:05.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.058+0000 7fb18179d640 1 -- start start 2024-02-06T22:18:05.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.058+0000 7fb18179d640 10 monclient: init 2024-02-06T22:18:05.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.059+0000 7fb18179d640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:05.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.059+0000 7fb18179d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-02-06T22:18:05.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.059+0000 7fb18179d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb17c0b5230 addr v1:172.21.15.135:6789/0 2024-02-06T22:18:05.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.059+0000 7fb18179d640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb17c13f960 0x7fb17c13fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.061 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.059+0000 7fb18179d640 10 monclient(hunting): picked mon.noname-a con 0x7fb17c13f960 addr v2:172.21.15.120:3300/0 2024-02-06T22:18:05.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.059+0000 7fb18179d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c1425f0 0x7fb17c142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.059+0000 7fb18179d640 10 monclient(hunting): picked mon.noname-b con 0x7fb17c1425f0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:05.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.059+0000 7fb18179d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17c0ab3e0 con 0x7fb17c0b5230 2024-02-06T22:18:05.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.059+0000 7fb18179d640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:05.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.059+0000 7fb18179d640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb17c0b5610 con 0x7fb17c13f960 2024-02-06T22:18:05.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.059+0000 7fb18179d640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:05.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.059+0000 7fb18179d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c0ab6c0 con 0x7fb17c1425f0 2024-02-06T22:18:05.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.059+0000 7fb18179d640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:05.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.059+0000 7fb18179d640 10 monclient(hunting): authenticate will time out at 3515.396105s 2024-02-06T22:18:05.062 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.062+0000 7fb173fff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb17c13f960 0x7fb17c13fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.063 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.063+0000 7fb17affd640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fb17c0b5230 0x7fb17c0a9850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.120:40114/0 (socket says 172.21.15.120:40114) 2024-02-06T22:18:05.063 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.063+0000 7fb17affd640 1 -- 172.21.15.120:0/1178678452 learned_addr learned my addr 172.21.15.120:0/1178678452 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T22:18:05.066 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.066+0000 7fb173fff640 10 monclient(hunting): get_auth_request con 0x7fb17c13f960 auth_method 0 2024-02-06T22:18:05.066 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.066+0000 7fb173fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:05.066 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.066+0000 7fb173fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:05.066 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.066+0000 7fb173fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:05.068 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.068+0000 7fb173fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:05.068 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.068+0000 7fb173fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:05.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.068+0000 7fb173fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:05.070 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.070+0000 7fb17b7fe640 1 --2- 172.21.15.120:0/1178678452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c1425f0 0x7fb17c142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.071+0000 7fb17b7fe640 10 monclient(hunting): get_auth_request con 0x7fb17c1425f0 auth_method 0 2024-02-06T22:18:05.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.071+0000 7fb17b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:05.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.071+0000 7fb17b7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:05.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.071+0000 7fb17b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:05.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.071+0000 7fb17b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:05.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.071+0000 7fb17b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:05.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.071+0000 7fb17b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:05.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.072+0000 7fb17a7fc640 1 -- 172.21.15.120:0/1178678452 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590899824 0 0) 0x7fb17c0ab3e0 con 0x7fb17c0b5230 2024-02-06T22:18:05.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.072+0000 7fb17a7fc640 10 monclient(hunting): _negotiate 2024-02-06T22:18:05.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.072+0000 7fb17a7fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:05.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.072+0000 7fb17a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:05.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.072+0000 7fb17a7fc640 10 monclient(hunting): my global_id is 21070 2024-02-06T22:18:05.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.072+0000 7fb17a7fc640 1 -- 172.21.15.120:0/1178678452 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb168003520 con 0x7fb17c0b5230 2024-02-06T22:18:05.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.073+0000 7fb17b7fe640 10 monclient(hunting): handle_auth_done global_id 18734 payload 274 2024-02-06T22:18:05.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.073+0000 7fb17b7fe640 1 -- 172.21.15.120:0/1178678452 >> v2:172.21.15.120:3300/0 conn(0x7fb17c13f960 msgr2=0x7fb17c13fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:05.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.073+0000 7fb17b7fe640 1 --2- 172.21.15.120:0/1178678452 >> v2:172.21.15.120:3300/0 conn(0x7fb17c13f960 0x7fb17c13fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.073+0000 7fb17b7fe640 1 -- 172.21.15.120:0/1178678452 >> v1:172.21.15.135:6789/0 conn(0x7fb17c0b5230 legacy=0x7fb17c0a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.073+0000 7fb17b7fe640 10 monclient: _finish_hunting 0 2024-02-06T22:18:05.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.073+0000 7fb17b7fe640 1 monclient: found mon.noname-b 2024-02-06T22:18:05.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.073+0000 7fb17b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:05.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.073+0000 7fb17b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T22:18:05.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.073+0000 7fb17b7fe640 1 -- 172.21.15.120:0/1178678452 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17c0aa040 con 0x7fb17c1425f0 2024-02-06T22:18:05.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.073+0000 7fb173fff640 1 --2- 172.21.15.120:0/1178678452 >> v2:172.21.15.120:3300/0 conn(0x7fb17c13f960 0x7fb17c13fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-06T22:18:05.076 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.076+0000 7fb17b7fe640 1 --2- 172.21.15.120:0/1178678452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c1425f0 0x7fb17c142a50 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7fb174009d90 tx=0x7fb17402bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b6be98de5b5ed4c server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:05.082 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb17a7fc640 1 -- 172.21.15.120:0/1178678452 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17402cb30 con 0x7fb17c1425f0 2024-02-06T22:18:05.082 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb17a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:05.082 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb17a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:18:05.082 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb17a7fc640 10 monclient: dump: 2024-02-06T22:18:05.082 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:05.082 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:05.082 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.082 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.082 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:05.083 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.083 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:05.083 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:05.083 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:05.083 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb17a7fc640 10 monclient: _finish_auth 0 2024-02-06T22:18:05.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb17a7fc640 10 monclient: _check_auth_tickets 2024-02-06T22:18:05.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb17a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:05.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb17a7fc640 1 -- 172.21.15.120:0/1178678452 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17402cc90 con 0x7fb17c1425f0 2024-02-06T22:18:05.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb18179d640 5 monclient: authenticate success, global_id 18734 2024-02-06T22:18:05.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb18179d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T22:18:05.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb17a7fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:05.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb18179d640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:05.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb18179d640 4 set_mon_vals no callback set 2024-02-06T22:18:05.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb17a7fc640 1 -- 172.21.15.120:0/1178678452 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb174034440 con 0x7fb17c1425f0 2024-02-06T22:18:05.084 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb17a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:05.084 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb17a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T22:18:05.084 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.081+0000 7fb17a7fc640 10 monclient: dump: 2024-02-06T22:18:05.084 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:05.084 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:05.084 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.084 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.084 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:05.084 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.084 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:05.084 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:05.084 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:05.084 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.086 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.082+0000 7fb18179d640 10 monclient: shutdown 2024-02-06T22:18:05.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.082+0000 7fb18179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:05.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.082+0000 7fb18179d640 1 -- 172.21.15.120:0/1178678452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c1425f0 msgr2=0x7fb17c142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.082+0000 7fb18179d640 1 --2- 172.21.15.120:0/1178678452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c1425f0 0x7fb17c142a50 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7fb174009d90 tx=0x7fb17402bd80 comp rx=0 tx=0).stop 2024-02-06T22:18:05.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.082+0000 7fb18179d640 1 -- 172.21.15.120:0/1178678452 shutdown_connections 2024-02-06T22:18:05.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.082+0000 7fb18179d640 1 --2- 172.21.15.120:0/1178678452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c1425f0 0x7fb17c142a50 unknown :-1 s=CLOSED pgs=3349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.082+0000 7fb18179d640 1 --2- 172.21.15.120:0/1178678452 >> v2:172.21.15.120:3300/0 conn(0x7fb17c13f960 0x7fb17c13fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.082+0000 7fb18179d640 1 -- 172.21.15.120:0/1178678452 >> 172.21.15.120:0/1178678452 conn(0x7fb17c06d1c0 msgr2=0x7fb17c06d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:05.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.082+0000 7fb18179d640 1 -- 172.21.15.120:0/1178678452 shutdown_connections 2024-02-06T22:18:05.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.082+0000 7fb18179d640 1 -- 172.21.15.120:0/1178678452 wait complete. 2024-02-06T22:18:05.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.083+0000 7fb18179d640 10 monclient: build_initial_monmap 2024-02-06T22:18:05.087 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.083+0000 7fb18179d640 10 monclient: monmap: 2024-02-06T22:18:05.087 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:05.087 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:05.087 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:05.088 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:05.088 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:05.088 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.088 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:05.088 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:05.088 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:05.088 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.083+0000 7fb18179d640 1 librados: starting msgr at 2024-02-06T22:18:05.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.083+0000 7fb18179d640 1 librados: starting objecter 2024-02-06T22:18:05.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.083+0000 7fb18179d640 1 Processor -- start 2024-02-06T22:18:05.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.083+0000 7fb18179d640 1 -- start start 2024-02-06T22:18:05.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.083+0000 7fb18179d640 1 librados: setting wanted keys 2024-02-06T22:18:05.088 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.083+0000 7fb18179d640 1 librados: calling monclient init 2024-02-06T22:18:05.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.083+0000 7fb18179d640 10 monclient: init 2024-02-06T22:18:05.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:05.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 10 monclient: _add_conns ranks=[0,2,1] 2024-02-06T22:18:05.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb17c0b5230 0x7fb17c13b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 10 monclient(hunting): picked mon.noname-a con 0x7fb17c0b5230 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:05.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c13f960 0x7fb17c13beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 10 monclient(hunting): picked mon.noname-c con 0x7fb17c13f960 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:05.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb17c1425f0 0x7fb17c138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 10 monclient(hunting): picked mon.noname-b con 0x7fb17c1425f0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:05.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:05.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c13c3f0 con 0x7fb17c0b5230 2024-02-06T22:18:05.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:05.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c0ac3b0 con 0x7fb17c1425f0 2024-02-06T22:18:05.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:05.089 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c0ac620 con 0x7fb17c13f960 2024-02-06T22:18:05.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:05.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.084+0000 7fb18179d640 10 monclient(hunting): authenticate will time out at 3515.420807s 2024-02-06T22:18:05.090 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.086+0000 7fb17b7fe640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb17c1425f0 0x7fb17c138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.087+0000 7fb17b7fe640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb17c1425f0 0x7fb17c138ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:51172/0 (socket says 172.21.15.120:51172) 2024-02-06T22:18:05.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.087+0000 7fb17b7fe640 1 -- 172.21.15.120:0/4244699782 learned_addr learned my addr 172.21.15.120:0/4244699782 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:05.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.087+0000 7fb17b7fe640 10 monclient(hunting): get_auth_request con 0x7fb17c1425f0 auth_method 0 2024-02-06T22:18:05.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.087+0000 7fb17b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:05.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.087+0000 7fb17b7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:05.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.087+0000 7fb17b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:05.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.087+0000 7fb173fff640 1 --2- 172.21.15.120:0/4244699782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c13f960 0x7fb17c13beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.087+0000 7fb173fff640 10 monclient(hunting): get_auth_request con 0x7fb17c13f960 auth_method 0 2024-02-06T22:18:05.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.087+0000 7fb173fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:05.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.087+0000 7fb173fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:05.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.087+0000 7fb173fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:05.092 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.088+0000 7fb173fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:05.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.088+0000 7fb173fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:05.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.088+0000 7fb173fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:05.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.088+0000 7fb17b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:05.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.088+0000 7fb17b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:05.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.088+0000 7fb17b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:05.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.088+0000 7fb17affd640 1 --2- 172.21.15.120:0/4244699782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb17c0b5230 0x7fb17c13b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.091+0000 7fb17affd640 10 monclient(hunting): get_auth_request con 0x7fb17c0b5230 auth_method 0 2024-02-06T22:18:05.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.091+0000 7fb17affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:05.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.091+0000 7fb17affd640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:05.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.091+0000 7fb17affd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:05.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.091+0000 7fb173fff640 10 monclient(hunting): handle_auth_done global_id 18737 payload 824 2024-02-06T22:18:05.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.091+0000 7fb173fff640 1 -- 172.21.15.120:0/4244699782 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb17c1425f0 msgr2=0x7fb17c138ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.091+0000 7fb173fff640 1 --2- 172.21.15.120:0/4244699782 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb17c1425f0 0x7fb17c138ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.091+0000 7fb173fff640 1 -- 172.21.15.120:0/4244699782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb17c0b5230 msgr2=0x7fb17c13b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.093 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.091+0000 7fb173fff640 1 --2- 172.21.15.120:0/4244699782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb17c0b5230 0x7fb17c13b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.091+0000 7fb173fff640 10 monclient: _finish_hunting 0 2024-02-06T22:18:05.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.091+0000 7fb173fff640 1 monclient: found mon.noname-c 2024-02-06T22:18:05.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.091+0000 7fb173fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:05.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.091+0000 7fb173fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.120:3301/0 2024-02-06T22:18:05.094 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.091+0000 7fb173fff640 1 -- 172.21.15.120:0/4244699782 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb164009400 con 0x7fb17c13f960 2024-02-06T22:18:05.094 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:44.909 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:05.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.095+0000 7fb173fff640 1 --2- 172.21.15.120:0/4244699782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c13f960 0x7fb17c13beb0 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 crypto rx=0x7fb16400e960 tx=0x7fb16400ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:05.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.095+0000 7fb1737fe640 10 client.?.objecter ms_handle_connect 0x7fb17c13f960 2024-02-06T22:18:05.096 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.095+0000 7fb1737fe640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:05.099 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:44.914 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:05.102 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb1737fe640 1 -- 172.21.15.120:0/4244699782 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16400ca00 con 0x7fb17c13f960 2024-02-06T22:18:05.102 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb1737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:05.102 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb1737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T22:18:05.102 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb1737fe640 10 monclient: dump: 2024-02-06T22:18:05.102 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:05.102 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:05.103 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.103 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.103 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:05.103 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.103 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:05.103 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:05.103 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:05.103 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb1737fe640 10 monclient: _finish_auth 0 2024-02-06T22:18:05.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb1737fe640 10 monclient: _check_auth_tickets 2024-02-06T22:18:05.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb1737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:05.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb1737fe640 1 -- 172.21.15.120:0/4244699782 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16400cb60 con 0x7fb17c13f960 2024-02-06T22:18:05.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb1737fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:05.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb1737fe640 1 -- 172.21.15.120:0/4244699782 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb164013470 con 0x7fb17c13f960 2024-02-06T22:18:05.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb1737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:05.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb1737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T22:18:05.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb1737fe640 10 monclient: dump: 2024-02-06T22:18:05.104 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:05.104 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:05.104 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.104 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.104 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:05.104 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.104 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:05.104 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:05.104 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:05.105 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.105 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb18179d640 5 monclient: authenticate success, global_id 18737 2024-02-06T22:18:05.105 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb18179d640 10 monclient: _renew_subs 2024-02-06T22:18:05.105 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb18179d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:18:05.105 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb18179d640 1 -- 172.21.15.120:0/4244699782 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb174009a70 con 0x7fb17c13f960 2024-02-06T22:18:05.105 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb179ffb640 4 set_mon_vals no callback set 2024-02-06T22:18:05.105 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb18179d640 10 client.18737.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:05.105 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb18179d640 10 monclient: _renew_subs 2024-02-06T22:18:05.105 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb18179d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:18:05.105 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb18179d640 1 -- 172.21.15.120:0/4244699782 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb17c13c830 con 0x7fb17c13f960 2024-02-06T22:18:05.105 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.101+0000 7fb18179d640 1 librados: init done 2024-02-06T22:18:05.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.102+0000 7fb18179d640 10 client.18737.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:05.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.102+0000 7fb18179d640 15 client.18737.objecter _session_command_op_assign -1 1 2024-02-06T22:18:05.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.102+0000 7fb18179d640 20 client.18737.objecter _get_session osd=-1 returning homeless 2024-02-06T22:18:05.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.102+0000 7fb18179d640 10 client.18737.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:05.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.102+0000 7fb18179d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.120:3301/0 2024-02-06T22:18:05.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.102+0000 7fb18179d640 1 -- 172.21.15.120:0/4244699782 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb17c14cea0 con 0x7fb17c13f960 2024-02-06T22:18:05.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.125+0000 7fb1737fe640 1 -- 172.21.15.120:0/4244699782 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb1640135d0 con 0x7fb17c13f960 2024-02-06T22:18:05.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.125+0000 7fb1737fe640 1 --2- 172.21.15.120:0/4244699782 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fb15c06d1a0 0x7fb15c06f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.126+0000 7fb17affd640 1 --2- 172.21.15.120:0/4244699782 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fb15c06d1a0 0x7fb15c06f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.126 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.126+0000 7fb17affd640 10 monclient: get_auth_request con 0x7fb15c06d1a0 auth_method 0 2024-02-06T22:18:05.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.126+0000 7fb17affd640 1 --2- 172.21.15.120:0/4244699782 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fb15c06d1a0 0x7fb15c06f660 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb16c005880 tx=0x7fb16c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:05.127 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.126+0000 7fb1737fe640 10 client.18737.objecter ms_handle_connect 0x7fb15c06d1a0 2024-02-06T22:18:05.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 1 -- 172.21.15.120:0/4244699782 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fb164058f20 con 0x7fb17c13f960 2024-02-06T22:18:05.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 10 client.18737.objecter ms_dispatch 0x7fb17c000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:05.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 3 client.18737.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:05.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 3 client.18737.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:05.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 10 client.18737.objecter checking command 1 2024-02-06T22:18:05.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 1 --2- 172.21.15.120:0/4244699782 >> [v2:172.21.15.135:6818/563193066,v1:172.21.15.135:6819/563193066] conn(0x7fb15c074f90 0x7fb15c077430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 20 client.18737.objecter _get_session s=0x7fb15c0748c0 osd=5 3 2024-02-06T22:18:05.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 20 client.18737.objecter put_session s=0x7fb15c0748c0 osd=5 3 2024-02-06T22:18:05.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 15 client.18737.objecter _session_command_op_remove -1 1 2024-02-06T22:18:05.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 20 client.18737.objecter _get_session s=0x7fb15c0748c0 osd=5 3 2024-02-06T22:18:05.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 20 client.18737.objecter get_session s=0x7fb15c0748c0 osd=5 3 2024-02-06T22:18:05.129 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 15 client.18737.objecter _session_command_op_assign 5 1 2024-02-06T22:18:05.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 20 client.18737.objecter put_session s=0x7fb15c0748c0 osd=5 4 2024-02-06T22:18:05.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 10 client.18737.objecter _send_command 1 2024-02-06T22:18:05.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 1 -- 172.21.15.120:0/4244699782 --> [v2:172.21.15.135:6818/563193066,v1:172.21.15.135:6819/563193066] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb15c077b00 con 0x7fb15c074f90 2024-02-06T22:18:05.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 20 client.18737.objecter dump_active .. 0 homeless 2024-02-06T22:18:05.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 1 -- 172.21.15.120:0/4244699782 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_get_version_reply(handle=1 version=1346) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb164009280 con 0x7fb17c13f960 2024-02-06T22:18:05.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.128+0000 7fb1737fe640 10 monclient: handle_get_version_reply finishing 1 version 1346 2024-02-06T22:18:05.130 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.129+0000 7fb17b7fe640 1 --2- 172.21.15.120:0/4244699782 >> [v2:172.21.15.135:6818/563193066,v1:172.21.15.135:6819/563193066] conn(0x7fb15c074f90 0x7fb15c077430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.131 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.129+0000 7fb17b7fe640 10 monclient: get_auth_request con 0x7fb15c074f90 auth_method 0 2024-02-06T22:18:05.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.132+0000 7fb17b7fe640 1 --2- 172.21.15.120:0/4244699782 >> [v2:172.21.15.135:6818/563193066,v1:172.21.15.135:6819/563193066] conn(0x7fb15c074f90 0x7fb15c077430 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:05.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.132+0000 7fb1737fe640 10 client.18737.objecter ms_handle_connect 0x7fb15c074f90 2024-02-06T22:18:05.133 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:44.947 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:05.156 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.156+0000 7fb1737fe640 1 -- 172.21.15.120:0/4244699782 <== osd.5 v2:172.21.15.135:6818/563193066 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb15c077b00 con 0x7fb15c074f90 2024-02-06T22:18:05.156 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.156+0000 7fb1737fe640 10 client.18737.objecter ms_dispatch 0x7fb17c000b90 command_reply(tid 1: 0 ) v1 2024-02-06T22:18:05.157 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.156+0000 7fb1737fe640 10 client.18737.objecter _finish_command 1 = system:0 2024-02-06T22:18:05.157 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.156+0000 7fb1737fe640 20 client.18737.objecter put_session s=0x7fb15c0748c0 osd=5 4 2024-02-06T22:18:05.157 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.156+0000 7fb1737fe640 15 client.18737.objecter _session_command_op_remove 5 1 2024-02-06T22:18:05.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.169+0000 7fb18179d640 10 client.18737.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-02-06T22:18:05.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.169+0000 7fb18179d640 15 client.18737.objecter _session_command_op_assign -1 2 2024-02-06T22:18:05.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.169+0000 7fb18179d640 20 client.18737.objecter _get_session s=0x7fb15c0748c0 osd=5 3 2024-02-06T22:18:05.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.169+0000 7fb18179d640 20 client.18737.objecter put_session s=0x7fb15c0748c0 osd=5 3 2024-02-06T22:18:05.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.169+0000 7fb18179d640 20 client.18737.objecter _get_session s=0x7fb15c0748c0 osd=5 3 2024-02-06T22:18:05.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.169+0000 7fb18179d640 15 client.18737.objecter _session_command_op_remove -1 2 2024-02-06T22:18:05.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.169+0000 7fb18179d640 20 client.18737.objecter get_session s=0x7fb15c0748c0 osd=5 3 2024-02-06T22:18:05.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.169+0000 7fb18179d640 15 client.18737.objecter _session_command_op_assign 5 2 2024-02-06T22:18:05.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.169+0000 7fb18179d640 20 client.18737.objecter put_session s=0x7fb15c0748c0 osd=5 4 2024-02-06T22:18:05.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.169+0000 7fb18179d640 10 client.18737.objecter _send_command 2 2024-02-06T22:18:05.170 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.169+0000 7fb18179d640 1 -- 172.21.15.120:0/4244699782 --> [v2:172.21.15.135:6818/563193066,v1:172.21.15.135:6819/563193066] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb17c0ac760 con 0x7fb15c074f90 2024-02-06T22:18:05.182 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb1737fe640 1 -- 172.21.15.120:0/4244699782 <== osd.5 v2:172.21.15.135:6818/563193066 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb17c0ac760 con 0x7fb15c074f90 2024-02-06T22:18:05.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb1737fe640 10 client.18737.objecter ms_dispatch 0x7fb17c000b90 command_reply(tid 2: 0 ) v1 2024-02-06T22:18:05.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb1737fe640 10 client.18737.objecter _finish_command 2 = system:0 2024-02-06T22:18:05.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb1737fe640 20 client.18737.objecter put_session s=0x7fb15c0748c0 osd=5 4 2024-02-06T22:18:05.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb1737fe640 15 client.18737.objecter _session_command_op_remove 5 2 2024-02-06T22:18:05.183 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-02-06T22:18:05.183 INFO:teuthology.orchestra.run.smithi120.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-06T22:18:05.183 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-02-06T22:18:05.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 10 librados: watch_flush enter 2024-02-06T22:18:05.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 10 librados: watch_flush exit 2024-02-06T22:18:05.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 10 client.18737.objecter close_session for osd.5 2024-02-06T22:18:05.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 1 -- 172.21.15.120:0/4244699782 >> [v2:172.21.15.135:6818/563193066,v1:172.21.15.135:6819/563193066] conn(0x7fb15c074f90 msgr2=0x7fb15c077430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 1 --2- 172.21.15.120:0/4244699782 >> [v2:172.21.15.135:6818/563193066,v1:172.21.15.135:6819/563193066] conn(0x7fb15c074f90 0x7fb15c077430 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 20 client.18737.objecter put_session s=0x7fb15c0748c0 osd=5 1 2024-02-06T22:18:05.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 20 client.18737.objecter shutdown clearing up homeless session... 2024-02-06T22:18:05.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 10 client.18737.objecter successfully canceled tick 2024-02-06T22:18:05.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 1 -- 172.21.15.120:0/4244699782 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fb15c06d1a0 msgr2=0x7fb15c06f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 1 --2- 172.21.15.120:0/4244699782 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fb15c06d1a0 0x7fb15c06f660 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb16c005880 tx=0x7fb16c009040 comp rx=0 tx=0).stop 2024-02-06T22:18:05.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 10 monclient: shutdown 2024-02-06T22:18:05.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:05.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 1 -- 172.21.15.120:0/4244699782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c13f960 msgr2=0x7fb17c13beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 1 --2- 172.21.15.120:0/4244699782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c13f960 0x7fb17c13beb0 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 crypto rx=0x7fb16400e960 tx=0x7fb16400ee20 comp rx=0 tx=0).stop 2024-02-06T22:18:05.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb17b7fe640 1 -- 172.21.15.120:0/4244699782 reap_dead start 2024-02-06T22:18:05.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 1 -- 172.21.15.120:0/4244699782 shutdown_connections 2024-02-06T22:18:05.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 1 -- 172.21.15.120:0/4244699782 >> 172.21.15.120:0/4244699782 conn(0x7fb17c06d1c0 msgr2=0x7fb17c0b8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:05.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 1 -- 172.21.15.120:0/4244699782 shutdown_connections 2024-02-06T22:18:05.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 1 -- 172.21.15.120:0/4244699782 wait complete. 2024-02-06T22:18:05.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.182+0000 7fb18179d640 1 librados: shutdown 2024-02-06T22:18:05.194 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-02-06T22:18:05.298 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.298+0000 7fcbf4a86640 10 monclient: get_monmap_and_config 2024-02-06T22:18:05.299 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.298+0000 7fcbf4a86640 10 monclient: build_initial_monmap 2024-02-06T22:18:05.299 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.298+0000 7fcbf4a86640 10 monclient: monmap: 2024-02-06T22:18:05.299 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:05.299 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:05.299 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:05.298778+0000 2024-02-06T22:18:05.299 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:05.298778+0000 2024-02-06T22:18:05.299 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:05.299 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.299 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:05.299 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:05.300 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:05.300 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:05.300 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:05.300 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.301 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 1 Processor -- start 2024-02-06T22:18:05.301 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 1 -- start start 2024-02-06T22:18:05.301 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 10 monclient: init 2024-02-06T22:18:05.301 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:05.301 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-02-06T22:18:05.302 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbf00b54b0 addr v1:172.21.15.120:6789/0 2024-02-06T22:18:05.302 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcbf013ff40 0x7fcbf00a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.302 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 10 monclient(hunting): picked mon.noname-a con 0x7fcbf013ff40 addr v2:172.21.15.120:3300/0 2024-02-06T22:18:05.302 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fcbf0142680 0x7fcbf0142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.302 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 10 monclient(hunting): picked mon.noname-c con 0x7fcbf0142680 addr v2:172.21.15.135:3300/0 2024-02-06T22:18:05.302 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf00ab720 con 0x7fcbf00b54b0 2024-02-06T22:18:05.302 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:05.302 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf00ab520 con 0x7fcbf013ff40 2024-02-06T22:18:05.302 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:05.302 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf00b5890 con 0x7fcbf0142680 2024-02-06T22:18:05.302 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:05.302 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.299+0000 7fcbf4a86640 10 monclient(hunting): authenticate will time out at 3515.636444s 2024-02-06T22:18:05.342 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.342+0000 7fcbeed9d640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fcbf0142680 0x7fcbf0142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.345 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.345+0000 7fcbee59c640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fcbf00b54b0 0x7fcbf013f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47502/0 (socket says 172.21.15.120:47502) 2024-02-06T22:18:05.345 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.345+0000 7fcbee59c640 1 -- 172.21.15.120:0/3766784235 learned_addr learned my addr 172.21.15.120:0/3766784235 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T22:18:05.347 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.347+0000 7fcbe7fff640 1 --2- 172.21.15.120:0/3766784235 >> v2:172.21.15.120:3300/0 conn(0x7fcbf013ff40 0x7fcbf00a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.348 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.348+0000 7fcbe7fff640 10 monclient(hunting): get_auth_request con 0x7fcbf013ff40 auth_method 0 2024-02-06T22:18:05.348 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.348+0000 7fcbe7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:05.348 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.348+0000 7fcbe7fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:05.348 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.348+0000 7fcbe7fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:05.349 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.348+0000 7fcbeed9d640 10 monclient(hunting): get_auth_request con 0x7fcbf0142680 auth_method 0 2024-02-06T22:18:05.349 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.348+0000 7fcbeed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:05.349 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.348+0000 7fcbeed9d640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:05.349 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.348+0000 7fcbeed9d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:05.349 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.349+0000 7fcbe7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:05.350 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.349+0000 7fcbe7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:05.350 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.349+0000 7fcbe7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:05.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.351+0000 7fcbeed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:05.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.351+0000 7fcbeed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:05.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.351+0000 7fcbeed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:05.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.353+0000 7fcbe7fff640 10 monclient(hunting): handle_auth_done global_id 16560 payload 274 2024-02-06T22:18:05.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.353+0000 7fcbe7fff640 1 -- 172.21.15.120:0/3766784235 >> v2:172.21.15.135:3300/0 conn(0x7fcbf0142680 msgr2=0x7fcbf0142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:05.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.353+0000 7fcbe7fff640 1 --2- 172.21.15.120:0/3766784235 >> v2:172.21.15.135:3300/0 conn(0x7fcbf0142680 0x7fcbf0142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.353+0000 7fcbe7fff640 1 -- 172.21.15.120:0/3766784235 >> v1:172.21.15.120:6789/0 conn(0x7fcbf00b54b0 legacy=0x7fcbf013f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.353+0000 7fcbe7fff640 10 monclient: _finish_hunting 0 2024-02-06T22:18:05.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.353+0000 7fcbe7fff640 1 monclient: found mon.noname-a 2024-02-06T22:18:05.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.353+0000 7fcbe7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:05.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.353+0000 7fcbe7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:18:05.354 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.353+0000 7fcbe7fff640 1 -- 172.21.15.120:0/3766784235 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf00b4800 con 0x7fcbf013ff40 2024-02-06T22:18:05.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.360+0000 7fcbe7fff640 1 --2- 172.21.15.120:0/3766784235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbf013ff40 0x7fcbf00a9860 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7fcbf00aa6a0 tx=0x7fcbd802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e0811a48f63b4e server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:05.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbedd9b640 1 -- 172.21.15.120:0/3766784235 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8031830 con 0x7fcbf013ff40 2024-02-06T22:18:05.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:05.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbedd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:18:05.373 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbedd9b640 10 monclient: dump: 2024-02-06T22:18:05.373 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:05.373 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:05.374 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.374 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.374 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:05.374 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.374 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:05.374 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:05.374 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:05.374 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbedd9b640 10 monclient: _finish_auth 0 2024-02-06T22:18:05.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbedd9b640 10 monclient: _check_auth_tickets 2024-02-06T22:18:05.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbedd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:05.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbedd9b640 1 -- 172.21.15.120:0/3766784235 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd8031990 con 0x7fcbf013ff40 2024-02-06T22:18:05.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbedd9b640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:05.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbedd9b640 1 -- 172.21.15.120:0/3766784235 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8031d20 con 0x7fcbf013ff40 2024-02-06T22:18:05.374 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:05.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbedd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:18:05.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbedd9b640 10 monclient: dump: 2024-02-06T22:18:05.375 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:05.375 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:05.375 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.375 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.375 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:05.375 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.375 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:05.375 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:05.375 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:05.375 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbf4a86640 5 monclient: authenticate success, global_id 16560 2024-02-06T22:18:05.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbf4a86640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:05.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.372+0000 7fcbf4a86640 4 set_mon_vals no callback set 2024-02-06T22:18:05.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.373+0000 7fcbf4a86640 10 monclient: shutdown 2024-02-06T22:18:05.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.373+0000 7fcbf4a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:05.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.373+0000 7fcbf4a86640 1 -- 172.21.15.120:0/3766784235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbf013ff40 msgr2=0x7fcbf00a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.373+0000 7fcbf4a86640 1 --2- 172.21.15.120:0/3766784235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbf013ff40 0x7fcbf00a9860 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7fcbf00aa6a0 tx=0x7fcbd802b8d0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.373+0000 7fcbf4a86640 1 -- 172.21.15.120:0/3766784235 shutdown_connections 2024-02-06T22:18:05.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.373+0000 7fcbf4a86640 1 --2- 172.21.15.120:0/3766784235 >> v2:172.21.15.135:3300/0 conn(0x7fcbf0142680 0x7fcbf0142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.373+0000 7fcbf4a86640 1 --2- 172.21.15.120:0/3766784235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbf013ff40 0x7fcbf00a9860 unknown :-1 s=CLOSED pgs=3355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.378 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.373+0000 7fcbf4a86640 1 -- 172.21.15.120:0/3766784235 >> 172.21.15.120:0/3766784235 conn(0x7fcbf00b8660 msgr2=0x7fcbf00b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:05.379 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.373+0000 7fcbf4a86640 1 -- 172.21.15.120:0/3766784235 shutdown_connections 2024-02-06T22:18:05.379 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.373+0000 7fcbf4a86640 1 -- 172.21.15.120:0/3766784235 wait complete. 2024-02-06T22:18:05.379 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.374+0000 7fcbf4a86640 10 monclient: build_initial_monmap 2024-02-06T22:18:05.379 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.374+0000 7fcbf4a86640 10 monclient: monmap: 2024-02-06T22:18:05.379 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:05.379 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:05.379 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:05.379 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:05.379 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:05.379 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.379 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:05.379 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:05.379 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:05.380 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 1 librados: starting msgr at 2024-02-06T22:18:05.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 1 librados: starting objecter 2024-02-06T22:18:05.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 1 Processor -- start 2024-02-06T22:18:05.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 1 -- start start 2024-02-06T22:18:05.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 1 librados: setting wanted keys 2024-02-06T22:18:05.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 1 librados: calling monclient init 2024-02-06T22:18:05.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 10 monclient: init 2024-02-06T22:18:05.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:05.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T22:18:05.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbf00b54b0 0x7fcbf0134d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 10 monclient(hunting): picked mon.noname-b con 0x7fcbf00b54b0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:05.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbf013ff40 0x7fcbf00b7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.380 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 10 monclient(hunting): picked mon.noname-a con 0x7fcbf013ff40 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:05.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbf0142680 0x7fcbf0135790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 10 monclient(hunting): picked mon.noname-c con 0x7fcbf0142680 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:05.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:05.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf01368f0 con 0x7fcbf013ff40 2024-02-06T22:18:05.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:05.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf01363d0 con 0x7fcbf00b54b0 2024-02-06T22:18:05.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:05.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0132060 con 0x7fcbf0142680 2024-02-06T22:18:05.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:05.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.375+0000 7fcbf4a86640 10 monclient(hunting): authenticate will time out at 3515.712370s 2024-02-06T22:18:05.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.376+0000 7fcbee59c640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbf00b54b0 0x7fcbf0134d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.376+0000 7fcbee59c640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbf00b54b0 0x7fcbf0134d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:51184/0 (socket says 172.21.15.120:51184) 2024-02-06T22:18:05.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.376+0000 7fcbee59c640 1 -- 172.21.15.120:0/1034022353 learned_addr learned my addr 172.21.15.120:0/1034022353 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:05.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.376+0000 7fcbee59c640 10 monclient(hunting): get_auth_request con 0x7fcbf00b54b0 auth_method 0 2024-02-06T22:18:05.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.376+0000 7fcbee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:05.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.376+0000 7fcbee59c640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:05.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.376+0000 7fcbee59c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:05.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.377+0000 7fcbee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:05.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.377+0000 7fcbee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:05.382 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.377+0000 7fcbee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:05.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.379+0000 7fcbee59c640 10 monclient(hunting): handle_auth_done global_id 21079 payload 824 2024-02-06T22:18:05.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.379+0000 7fcbee59c640 1 -- 172.21.15.120:0/1034022353 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbf0142680 msgr2=0x7fcbf0135790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.379+0000 7fcbee59c640 1 --2- 172.21.15.120:0/1034022353 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbf0142680 0x7fcbf0135790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.379+0000 7fcbee59c640 1 -- 172.21.15.120:0/1034022353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbf013ff40 msgr2=0x7fcbf00b7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.379+0000 7fcbee59c640 1 --2- 172.21.15.120:0/1034022353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbf013ff40 0x7fcbf00b7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.379+0000 7fcbee59c640 10 monclient: _finish_hunting 0 2024-02-06T22:18:05.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.379+0000 7fcbee59c640 1 monclient: found mon.noname-b 2024-02-06T22:18:05.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.379+0000 7fcbee59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:05.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.379+0000 7fcbee59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T22:18:05.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.379+0000 7fcbee59c640 1 -- 172.21.15.120:0/1034022353 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd8009400 con 0x7fcbf00b54b0 2024-02-06T22:18:05.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.381+0000 7fcbee59c640 1 --2- 172.21.15.120:0/1034022353 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbf00b54b0 0x7fcbf0134d40 secure :-1 s=READY pgs=5098 cs=0 l=1 rev1=1 crypto rx=0x7fcbe000c3b0 tx=0x7fcbe000c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:05.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.381+0000 7fcbe6ffd640 10 client.?.objecter ms_handle_connect 0x7fcbf00b54b0 2024-02-06T22:18:05.384 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.381+0000 7fcbe6ffd640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:05.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbe6ffd640 1 -- 172.21.15.120:0/1034022353 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0005ce0 con 0x7fcbf00b54b0 2024-02-06T22:18:05.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:05.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbe6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:18:05.386 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbe6ffd640 10 monclient: dump: 2024-02-06T22:18:05.386 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:05.386 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:05.386 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.386 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.386 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:05.386 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.386 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:05.386 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:05.386 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:05.387 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbe6ffd640 10 monclient: _finish_auth 0 2024-02-06T22:18:05.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbe6ffd640 10 monclient: _check_auth_tickets 2024-02-06T22:18:05.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbe6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:05.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbe6ffd640 1 -- 172.21.15.120:0/1034022353 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe0005e40 con 0x7fcbf00b54b0 2024-02-06T22:18:05.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbf4a86640 5 monclient: authenticate success, global_id 21079 2024-02-06T22:18:05.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbf4a86640 10 monclient: _renew_subs 2024-02-06T22:18:05.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbf4a86640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:05.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbf4a86640 1 -- 172.21.15.120:0/1034022353 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf0132280 con 0x7fcbf00b54b0 2024-02-06T22:18:05.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbe6ffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:05.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbe6ffd640 1 -- 172.21.15.120:0/1034022353 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe00072f0 con 0x7fcbf00b54b0 2024-02-06T22:18:05.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:05.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbe6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T22:18:05.387 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbe6ffd640 10 monclient: dump: 2024-02-06T22:18:05.387 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:05.388 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:05.388 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.388 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.388 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:05.388 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.388 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:05.388 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:05.388 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:05.388 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.388 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbed59a640 4 set_mon_vals no callback set 2024-02-06T22:18:05.388 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbf4a86640 10 client.21079.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:05.388 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbf4a86640 10 monclient: _renew_subs 2024-02-06T22:18:05.388 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbf4a86640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:05.388 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbf4a86640 1 -- 172.21.15.120:0/1034022353 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf01327b0 con 0x7fcbf00b54b0 2024-02-06T22:18:05.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.385+0000 7fcbf4a86640 1 librados: init done 2024-02-06T22:18:05.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.386+0000 7fcbf4a86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:05.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.386+0000 7fcbf4a86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:05.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.386+0000 7fcbf4a86640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:05.389 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.386+0000 7fcbf4a86640 1 -- 172.21.15.120:0/1034022353 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf004ef90 con 0x7fcbf00b54b0 2024-02-06T22:18:05.626 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.625+0000 7fcbe6ffd640 1 -- 172.21.15.120:0/1034022353 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcbe0007510 con 0x7fcbf00b54b0 2024-02-06T22:18:05.626 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.625+0000 7fcbe6ffd640 1 --2- 172.21.15.120:0/1034022353 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fcbd006d1a0 0x7fcbd006f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.626 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.626+0000 7fcbe7fff640 1 --2- 172.21.15.120:0/1034022353 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fcbd006d1a0 0x7fcbd006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.626 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.626+0000 7fcbe7fff640 10 monclient: get_auth_request con 0x7fcbd006d1a0 auth_method 0 2024-02-06T22:18:05.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.626+0000 7fcbe6ffd640 10 client.21079.objecter ms_handle_connect 0x7fcbd006d1a0 2024-02-06T22:18:05.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.626+0000 7fcbe7fff640 1 --2- 172.21.15.120:0/1034022353 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fcbd006d1a0 0x7fcbd006f660 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8036000 tx=0x7fcbd8030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:05.630 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.629+0000 7fcbe6ffd640 1 -- 172.21.15.120:0/1034022353 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fcbe0058a10 con 0x7fcbf00b54b0 2024-02-06T22:18:05.630 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.629+0000 7fcbe6ffd640 10 client.21079.objecter ms_dispatch 0x7fcbf0000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:05.630 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.629+0000 7fcbe6ffd640 3 client.21079.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:05.630 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.629+0000 7fcbe6ffd640 3 client.21079.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:05.630 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.630+0000 7fcbe6ffd640 20 client.21079.objecter dump_active .. 0 homeless 2024-02-06T22:18:05.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.639+0000 7fcbe6ffd640 1 -- 172.21.15.120:0/1034022353 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcbe00545b0 con 0x7fcbf00b54b0 2024-02-06T22:18:05.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.639+0000 7fcbe6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:05.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.639+0000 7fcbe6ffd640 10 monclient: _finish_command 1 = system:0 2024-02-06T22:18:05.853 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.853+0000 7fcbf4a86640 1 -- 172.21.15.120:0/1034022353 --> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbf00aa450 con 0x7fcbd006d1a0 2024-02-06T22:18:05.854 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.853+0000 7fcbe6ffd640 1 -- 172.21.15.120:0/1034022353 <== mgr.4100 v2:172.21.15.135:6800/315664144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcbd8031630 con 0x7fcbd006d1a0 2024-02-06T22:18:05.854 INFO:teuthology.orchestra.run.smithi120.stderr:instructed osd(s) 5 to deep-scrub 2024-02-06T22:18:05.856 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 10 librados: watch_flush enter 2024-02-06T22:18:05.857 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 10 librados: watch_flush exit 2024-02-06T22:18:05.857 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 20 client.21079.objecter shutdown clearing up homeless session... 2024-02-06T22:18:05.857 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 10 client.21079.objecter successfully canceled tick 2024-02-06T22:18:05.857 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 1 -- 172.21.15.120:0/1034022353 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fcbd006d1a0 msgr2=0x7fcbd006f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.857 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 1 --2- 172.21.15.120:0/1034022353 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fcbd006d1a0 0x7fcbd006f660 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8036000 tx=0x7fcbd8030000 comp rx=0 tx=0).stop 2024-02-06T22:18:05.857 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 10 monclient: shutdown 2024-02-06T22:18:05.857 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:05.857 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 1 -- 172.21.15.120:0/1034022353 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbf00b54b0 msgr2=0x7fcbf0134d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.857 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 1 --2- 172.21.15.120:0/1034022353 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbf00b54b0 0x7fcbf0134d40 secure :-1 s=READY pgs=5098 cs=0 l=1 rev1=1 crypto rx=0x7fcbe000c3b0 tx=0x7fcbe000c870 comp rx=0 tx=0).stop 2024-02-06T22:18:05.857 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 1 -- 172.21.15.120:0/1034022353 shutdown_connections 2024-02-06T22:18:05.857 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 1 --2- 172.21.15.120:0/1034022353 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fcbd006d1a0 0x7fcbd006f660 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.857 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 1 --2- 172.21.15.120:0/1034022353 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbf0142680 0x7fcbf0135790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.857 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 1 --2- 172.21.15.120:0/1034022353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbf013ff40 0x7fcbf00b7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.858 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 1 --2- 172.21.15.120:0/1034022353 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbf00b54b0 0x7fcbf0134d40 unknown :-1 s=CLOSED pgs=5098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.858 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.855+0000 7fcbf4a86640 1 -- 172.21.15.120:0/1034022353 >> 172.21.15.120:0/1034022353 conn(0x7fcbf00b8660 msgr2=0x7fcbf013f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:05.858 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.856+0000 7fcbf4a86640 1 -- 172.21.15.120:0/1034022353 shutdown_connections 2024-02-06T22:18:05.858 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.856+0000 7fcbf4a86640 1 -- 172.21.15.120:0/1034022353 wait complete. 2024-02-06T22:18:05.858 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.856+0000 7fcbf4a86640 1 librados: shutdown 2024-02-06T22:18:05.868 INFO:tasks.ceph:Scrubbing osd.6 2024-02-06T22:18:05.868 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-02-06T22:18:05.973 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.973+0000 7fbc0b40e640 10 monclient: get_monmap_and_config 2024-02-06T22:18:05.974 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.973+0000 7fbc0b40e640 10 monclient: build_initial_monmap 2024-02-06T22:18:05.974 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.973+0000 7fbc0b40e640 10 monclient: monmap: 2024-02-06T22:18:05.974 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:05.974 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:05.974 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:05.973446+0000 2024-02-06T22:18:05.974 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:05.973446+0000 2024-02-06T22:18:05.974 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:05.974 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.974 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:05.974 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:05.974 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:05.974 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:05.974 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:05.975 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.973+0000 7fbc0b40e640 1 Processor -- start 2024-02-06T22:18:05.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 1 -- start start 2024-02-06T22:18:05.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 10 monclient: init 2024-02-06T22:18:05.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:05.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-02-06T22:18:05.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbc0406c7e0 0x7fbc04142fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0406c7e0 addr v2:172.21.15.120:3300/0 2024-02-06T22:18:05.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc0406cbc0 addr v1:172.21.15.120:6789/0 2024-02-06T22:18:05.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbc0413f8b0 0x7fbc0413fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 10 monclient(hunting): picked mon.noname-c con 0x7fbc0413f8b0 addr v2:172.21.15.135:3300/0 2024-02-06T22:18:05.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc04140270 con 0x7fbc0406cbc0 2024-02-06T22:18:05.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:05.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc041434f0 con 0x7fbc0406c7e0 2024-02-06T22:18:05.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:05.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc04142ac0 con 0x7fbc0413f8b0 2024-02-06T22:18:05.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:05.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.974+0000 7fbc0b40e640 10 monclient(hunting): authenticate will time out at 3516.311139s 2024-02-06T22:18:05.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.975+0000 7fbc091aa640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbc0406c7e0 0x7fbc04142fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.975+0000 7fbc089a9640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fbc0406cbc0 0x7fbc040b0980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47510/0 (socket says 172.21.15.120:47510) 2024-02-06T22:18:05.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.975+0000 7fbc089a9640 1 -- 172.21.15.120:0/3398128471 learned_addr learned my addr 172.21.15.120:0/3398128471 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T22:18:05.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.975+0000 7fbc099ab640 1 --2- 172.21.15.120:0/3398128471 >> v2:172.21.15.135:3300/0 conn(0x7fbc0413f8b0 0x7fbc0413fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.975+0000 7fbc099ab640 10 monclient(hunting): get_auth_request con 0x7fbc0413f8b0 auth_method 0 2024-02-06T22:18:05.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.975+0000 7fbc099ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:05.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.975+0000 7fbc099ab640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:05.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.975+0000 7fbc099ab640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:05.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.976+0000 7fbc091aa640 10 monclient(hunting): get_auth_request con 0x7fbc0406c7e0 auth_method 0 2024-02-06T22:18:05.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.976+0000 7fbc091aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:05.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.976+0000 7fbc091aa640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:05.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.976+0000 7fbc091aa640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:05.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.976+0000 7fbc099ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:05.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.976+0000 7fbc099ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:05.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.976+0000 7fbc099ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:05.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.976+0000 7fbc091aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:05.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.976+0000 7fbc091aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:05.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.976+0000 7fbc091aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:05.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.977+0000 7fbc099ab640 10 monclient(hunting): handle_auth_done global_id 21082 payload 274 2024-02-06T22:18:05.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.977+0000 7fbc099ab640 1 -- 172.21.15.120:0/3398128471 >> v2:172.21.15.120:3300/0 conn(0x7fbc0406c7e0 msgr2=0x7fbc04142fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:05.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.977+0000 7fbc099ab640 1 --2- 172.21.15.120:0/3398128471 >> v2:172.21.15.120:3300/0 conn(0x7fbc0406c7e0 0x7fbc04142fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.977+0000 7fbc099ab640 1 -- 172.21.15.120:0/3398128471 >> v1:172.21.15.120:6789/0 conn(0x7fbc0406cbc0 legacy=0x7fbc040b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.977+0000 7fbc099ab640 10 monclient: _finish_hunting 0 2024-02-06T22:18:05.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.977+0000 7fbc099ab640 1 monclient: found mon.noname-c 2024-02-06T22:18:05.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.977+0000 7fbc099ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:05.980 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.977+0000 7fbc099ab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T22:18:05.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.977+0000 7fbc099ab640 1 -- 172.21.15.120:0/3398128471 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0406d580 con 0x7fbc0413f8b0 2024-02-06T22:18:05.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.979+0000 7fbc099ab640 1 --2- 172.21.15.120:0/3398128471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc0413f8b0 0x7fbc0413fd30 secure :-1 s=READY pgs=5099 cs=0 l=1 rev1=1 crypto rx=0x7fbc040b2440 tx=0x7fbbfc02be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa9f203b5f8196e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:05.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbbfbfff640 1 -- 172.21.15.120:0/3398128471 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc02cbf0 con 0x7fbc0413f8b0 2024-02-06T22:18:05.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbbfbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:05.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbbfbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T22:18:05.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbbfbfff640 10 monclient: dump: 2024-02-06T22:18:05.984 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:05.984 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:05.984 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.984 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.984 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:05.984 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.984 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:05.984 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:05.984 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:05.985 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbbfbfff640 10 monclient: _finish_auth 0 2024-02-06T22:18:05.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbbfbfff640 10 monclient: _check_auth_tickets 2024-02-06T22:18:05.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbbfbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:05.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbbfbfff640 1 -- 172.21.15.120:0/3398128471 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbfc02cd50 con 0x7fbc0413f8b0 2024-02-06T22:18:05.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbbfbfff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:05.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbbfbfff640 1 -- 172.21.15.120:0/3398128471 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc034660 con 0x7fbc0413f8b0 2024-02-06T22:18:05.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbbfbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:05.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbbfbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T22:18:05.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbbfbfff640 10 monclient: dump: 2024-02-06T22:18:05.985 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:05.985 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:05.985 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.985 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.985 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:05.986 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.986 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:05.986 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:05.986 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:05.986 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbc0b40e640 5 monclient: authenticate success, global_id 21082 2024-02-06T22:18:05.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbc0b40e640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:05.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.983+0000 7fbc0b40e640 4 set_mon_vals no callback set 2024-02-06T22:18:05.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.984+0000 7fbc0b40e640 10 monclient: shutdown 2024-02-06T22:18:05.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.984+0000 7fbc0b40e640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:05.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.984+0000 7fbc0b40e640 1 -- 172.21.15.120:0/3398128471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc0413f8b0 msgr2=0x7fbc0413fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.984+0000 7fbc0b40e640 1 --2- 172.21.15.120:0/3398128471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc0413f8b0 0x7fbc0413fd30 secure :-1 s=READY pgs=5099 cs=0 l=1 rev1=1 crypto rx=0x7fbc040b2440 tx=0x7fbbfc02be40 comp rx=0 tx=0).stop 2024-02-06T22:18:05.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.984+0000 7fbc0b40e640 1 -- 172.21.15.120:0/3398128471 shutdown_connections 2024-02-06T22:18:05.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.984+0000 7fbc0b40e640 1 --2- 172.21.15.120:0/3398128471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc0413f8b0 0x7fbc0413fd30 unknown :-1 s=CLOSED pgs=5099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.984+0000 7fbc0b40e640 1 --2- 172.21.15.120:0/3398128471 >> v2:172.21.15.120:3300/0 conn(0x7fbc0406c7e0 0x7fbc04142fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.984+0000 7fbc0b40e640 1 -- 172.21.15.120:0/3398128471 >> 172.21.15.120:0/3398128471 conn(0x7fbc040b6160 msgr2=0x7fbc040b6570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:05.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.984+0000 7fbc0b40e640 1 -- 172.21.15.120:0/3398128471 shutdown_connections 2024-02-06T22:18:05.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.984+0000 7fbc0b40e640 1 -- 172.21.15.120:0/3398128471 wait complete. 2024-02-06T22:18:05.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.985+0000 7fbc0b40e640 10 monclient: build_initial_monmap 2024-02-06T22:18:05.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.985+0000 7fbc0b40e640 10 monclient: monmap: 2024-02-06T22:18:05.991 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:05.991 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:05.991 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:05.991 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:05.991 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:05.991 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.991 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:05.991 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:05.992 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:05.992 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.985+0000 7fbc0b40e640 1 librados: starting msgr at 2024-02-06T22:18:05.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.985+0000 7fbc0b40e640 1 librados: starting objecter 2024-02-06T22:18:05.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.985+0000 7fbc0b40e640 1 Processor -- start 2024-02-06T22:18:05.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.985+0000 7fbc0b40e640 1 -- start start 2024-02-06T22:18:05.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.985+0000 7fbc0b40e640 1 librados: setting wanted keys 2024-02-06T22:18:05.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.985+0000 7fbc0b40e640 1 librados: calling monclient init 2024-02-06T22:18:05.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.985+0000 7fbc0b40e640 10 monclient: init 2024-02-06T22:18:05.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.985+0000 7fbc0b40e640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:05.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc0b40e640 10 monclient: _add_conns ranks=[1,2,0] 2024-02-06T22:18:05.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc0b40e640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc0406c7e0 0x7fbc04141a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc0b40e640 10 monclient(hunting): picked mon.noname-b con 0x7fbc0406c7e0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:05.992 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc0b40e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc0406cbc0 0x7fbc041dab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc0b40e640 10 monclient(hunting): picked mon.noname-c con 0x7fbc0406cbc0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:05.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc0b40e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc0413f8b0 0x7fbc041db060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:05.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc0b40e640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0413f8b0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:05.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc0b40e640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:05.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc0b40e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc041db5a0 con 0x7fbc0413f8b0 2024-02-06T22:18:05.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc0b40e640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:05.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc0b40e640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc041db790 con 0x7fbc0406c7e0 2024-02-06T22:18:05.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc0b40e640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:05.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc0b40e640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc04144f40 con 0x7fbc0406cbc0 2024-02-06T22:18:05.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc0b40e640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:05.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc0b40e640 10 monclient(hunting): authenticate will time out at 3516.322612s 2024-02-06T22:18:05.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc099ab640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc0413f8b0 0x7fbc041db060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc089a9640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc0406cbc0 0x7fbc041dab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc099ab640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc0413f8b0 0x7fbc041db060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:49674/0 (socket says 172.21.15.120:49674) 2024-02-06T22:18:05.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc099ab640 1 -- 172.21.15.120:0/263725513 learned_addr learned my addr 172.21.15.120:0/263725513 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:05.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc099ab640 10 monclient(hunting): get_auth_request con 0x7fbc0413f8b0 auth_method 0 2024-02-06T22:18:05.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc099ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:05.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc099ab640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:05.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.986+0000 7fbc099ab640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:05.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc091aa640 1 --2- 172.21.15.120:0/263725513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc0406c7e0 0x7fbc04141a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:05.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc089a9640 10 monclient(hunting): get_auth_request con 0x7fbc0406cbc0 auth_method 0 2024-02-06T22:18:05.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc089a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:05.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc089a9640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:05.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc089a9640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:05.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc091aa640 10 monclient(hunting): get_auth_request con 0x7fbc0406c7e0 auth_method 0 2024-02-06T22:18:05.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc091aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:05.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc091aa640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:05.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc091aa640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:05.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc099ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:05.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc099ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:05.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc099ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:05.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc091aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:05.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc091aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:05.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc091aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:05.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc089a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:05.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc089a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:05.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.987+0000 7fbc089a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:05.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.990+0000 7fbc099ab640 10 monclient(hunting): handle_auth_done global_id 16569 payload 824 2024-02-06T22:18:05.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.990+0000 7fbc099ab640 1 -- 172.21.15.120:0/263725513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc0406cbc0 msgr2=0x7fbc041dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.990+0000 7fbc099ab640 1 --2- 172.21.15.120:0/263725513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc0406cbc0 0x7fbc041dab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.990+0000 7fbc099ab640 1 -- 172.21.15.120:0/263725513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc0406c7e0 msgr2=0x7fbc04141a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:05.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.990+0000 7fbc099ab640 1 --2- 172.21.15.120:0/263725513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc0406c7e0 0x7fbc04141a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:05.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.990+0000 7fbc099ab640 10 monclient: _finish_hunting 0 2024-02-06T22:18:05.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.990+0000 7fbc099ab640 1 monclient: found mon.noname-a 2024-02-06T22:18:05.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.990+0000 7fbc099ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:05.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.990+0000 7fbc099ab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:18:05.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.990+0000 7fbc099ab640 1 -- 172.21.15.120:0/263725513 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbfc009a70 con 0x7fbc0413f8b0 2024-02-06T22:18:05.997 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.992+0000 7fbc099ab640 1 --2- 172.21.15.120:0/263725513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc0413f8b0 0x7fbc041db060 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009e80 tx=0x7fbbfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:05.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.992+0000 7fbbf9ffb640 10 client.?.objecter ms_handle_connect 0x7fbc0413f8b0 2024-02-06T22:18:05.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.992+0000 7fbbf9ffb640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:05.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.995+0000 7fbbf9ffb640 1 -- 172.21.15.120:0/263725513 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc040650 con 0x7fbc0413f8b0 2024-02-06T22:18:05.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.995+0000 7fbbf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:05.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbbf9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:18:05.998 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbbf9ffb640 10 monclient: dump: 2024-02-06T22:18:05.998 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:05.998 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:05.998 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.998 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:05.999 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:05.999 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:05.999 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:05.999 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:05.999 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:05.999 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:05.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbbf9ffb640 10 monclient: _finish_auth 0 2024-02-06T22:18:05.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbbf9ffb640 10 monclient: _check_auth_tickets 2024-02-06T22:18:05.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbbf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:05.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbbf9ffb640 1 -- 172.21.15.120:0/263725513 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbfc0407b0 con 0x7fbc0413f8b0 2024-02-06T22:18:05.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbbf9ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:05.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbbf9ffb640 1 -- 172.21.15.120:0/263725513 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc040b60 con 0x7fbc0413f8b0 2024-02-06T22:18:05.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbc0b40e640 5 monclient: authenticate success, global_id 16569 2024-02-06T22:18:05.999 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbc0b40e640 10 monclient: _renew_subs 2024-02-06T22:18:06.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbc0b40e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:06.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbc0b40e640 1 -- 172.21.15.120:0/263725513 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc04145160 con 0x7fbc0413f8b0 2024-02-06T22:18:06.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbbfb7fe640 4 set_mon_vals no callback set 2024-02-06T22:18:06.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbbf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:06.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbbf9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:18:06.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbbf9ffb640 10 monclient: dump: 2024-02-06T22:18:06.000 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:06.000 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:06.000 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.000 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.000 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:06.000 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.000 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:06.000 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:06.001 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:06.001 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:06.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbc0b40e640 10 client.16569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:06.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbc0b40e640 10 monclient: _renew_subs 2024-02-06T22:18:06.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbc0b40e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:06.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbc0b40e640 1 -- 172.21.15.120:0/263725513 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc04145610 con 0x7fbc0413f8b0 2024-02-06T22:18:06.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.996+0000 7fbc0b40e640 1 librados: init done 2024-02-06T22:18:06.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.997+0000 7fbc0b40e640 10 client.16569.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:06.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.997+0000 7fbc0b40e640 15 client.16569.objecter _session_command_op_assign -1 1 2024-02-06T22:18:06.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.997+0000 7fbc0b40e640 20 client.16569.objecter _get_session osd=-1 returning homeless 2024-02-06T22:18:06.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.997+0000 7fbc0b40e640 10 client.16569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:06.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.997+0000 7fbc0b40e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:06.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:05.997+0000 7fbc0b40e640 1 -- 172.21.15.120:0/263725513 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbc0404ef90 con 0x7fbc0413f8b0 2024-02-06T22:18:06.016 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.016+0000 7fbbf9ffb640 1 -- 172.21.15.120:0/263725513 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbbfc040cc0 con 0x7fbc0413f8b0 2024-02-06T22:18:06.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.017+0000 7fbbf9ffb640 1 --2- 172.21.15.120:0/263725513 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbbe806d220 0x7fbbe806f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.017+0000 7fbc091aa640 1 --2- 172.21.15.120:0/263725513 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbbe806d220 0x7fbbe806f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.017+0000 7fbc091aa640 10 monclient: get_auth_request con 0x7fbbe806d220 auth_method 0 2024-02-06T22:18:06.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.017+0000 7fbbf9ffb640 10 client.16569.objecter ms_handle_connect 0x7fbbe806d220 2024-02-06T22:18:06.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.017+0000 7fbc091aa640 1 --2- 172.21.15.120:0/263725513 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbbe806d220 0x7fbbe806f6e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fbc00009b40 tx=0x7fbc00009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:06.020 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 1 -- 172.21.15.120:0/263725513 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fbbfc07bcd0 con 0x7fbc0413f8b0 2024-02-06T22:18:06.020 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 10 client.16569.objecter ms_dispatch 0x7fbc04000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:06.020 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 3 client.16569.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:06.020 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 3 client.16569.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:06.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 10 client.16569.objecter checking command 1 2024-02-06T22:18:06.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 1 --2- 172.21.15.120:0/263725513 >> [v2:172.21.15.135:6810/1080459988,v1:172.21.15.135:6811/1080459988] conn(0x7fbbe8074fb0 0x7fbbe8077450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 20 client.16569.objecter _get_session s=0x7fbbe80748e0 osd=6 3 2024-02-06T22:18:06.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 20 client.16569.objecter put_session s=0x7fbbe80748e0 osd=6 3 2024-02-06T22:18:06.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 15 client.16569.objecter _session_command_op_remove -1 1 2024-02-06T22:18:06.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 20 client.16569.objecter _get_session s=0x7fbbe80748e0 osd=6 3 2024-02-06T22:18:06.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 20 client.16569.objecter get_session s=0x7fbbe80748e0 osd=6 3 2024-02-06T22:18:06.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 15 client.16569.objecter _session_command_op_assign 6 1 2024-02-06T22:18:06.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 20 client.16569.objecter put_session s=0x7fbbe80748e0 osd=6 4 2024-02-06T22:18:06.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 10 client.16569.objecter _send_command 1 2024-02-06T22:18:06.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 1 -- 172.21.15.120:0/263725513 --> [v2:172.21.15.135:6810/1080459988,v1:172.21.15.135:6811/1080459988] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbe8077b20 con 0x7fbbe8074fb0 2024-02-06T22:18:06.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 20 client.16569.objecter dump_active .. 0 homeless 2024-02-06T22:18:06.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 1 -- 172.21.15.120:0/263725513 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_get_version_reply(handle=1 version=1346) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbfc077f70 con 0x7fbc0413f8b0 2024-02-06T22:18:06.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.019+0000 7fbbf9ffb640 10 monclient: handle_get_version_reply finishing 1 version 1346 2024-02-06T22:18:06.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.020+0000 7fbc089a9640 1 --2- 172.21.15.120:0/263725513 >> [v2:172.21.15.135:6810/1080459988,v1:172.21.15.135:6811/1080459988] conn(0x7fbbe8074fb0 0x7fbbe8077450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.020+0000 7fbc089a9640 10 monclient: get_auth_request con 0x7fbbe8074fb0 auth_method 0 2024-02-06T22:18:06.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.023+0000 7fbc089a9640 1 --2- 172.21.15.120:0/263725513 >> [v2:172.21.15.135:6810/1080459988,v1:172.21.15.135:6811/1080459988] conn(0x7fbbe8074fb0 0x7fbbe8077450 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:06.023 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.023+0000 7fbbf9ffb640 10 client.16569.objecter ms_handle_connect 0x7fbbe8074fb0 2024-02-06T22:18:06.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.046+0000 7fbbf9ffb640 1 -- 172.21.15.120:0/263725513 <== osd.6 v2:172.21.15.135:6810/1080459988 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbbe8077b20 con 0x7fbbe8074fb0 2024-02-06T22:18:06.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.046+0000 7fbbf9ffb640 10 client.16569.objecter ms_dispatch 0x7fbc04000b90 command_reply(tid 1: 0 ) v1 2024-02-06T22:18:06.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.046+0000 7fbbf9ffb640 10 client.16569.objecter _finish_command 1 = system:0 2024-02-06T22:18:06.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.046+0000 7fbbf9ffb640 20 client.16569.objecter put_session s=0x7fbbe80748e0 osd=6 4 2024-02-06T22:18:06.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.046+0000 7fbbf9ffb640 15 client.16569.objecter _session_command_op_remove 6 1 2024-02-06T22:18:06.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.058+0000 7fbc0b40e640 10 client.16569.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-02-06T22:18:06.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.058+0000 7fbc0b40e640 15 client.16569.objecter _session_command_op_assign -1 2 2024-02-06T22:18:06.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.058+0000 7fbc0b40e640 20 client.16569.objecter _get_session s=0x7fbbe80748e0 osd=6 3 2024-02-06T22:18:06.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.058+0000 7fbc0b40e640 20 client.16569.objecter put_session s=0x7fbbe80748e0 osd=6 3 2024-02-06T22:18:06.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.058+0000 7fbc0b40e640 20 client.16569.objecter _get_session s=0x7fbbe80748e0 osd=6 3 2024-02-06T22:18:06.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.058+0000 7fbc0b40e640 15 client.16569.objecter _session_command_op_remove -1 2 2024-02-06T22:18:06.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.058+0000 7fbc0b40e640 20 client.16569.objecter get_session s=0x7fbbe80748e0 osd=6 3 2024-02-06T22:18:06.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.058+0000 7fbc0b40e640 15 client.16569.objecter _session_command_op_assign 6 2 2024-02-06T22:18:06.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.058+0000 7fbc0b40e640 20 client.16569.objecter put_session s=0x7fbbe80748e0 osd=6 4 2024-02-06T22:18:06.059 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.058+0000 7fbc0b40e640 10 client.16569.objecter _send_command 2 2024-02-06T22:18:06.060 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.058+0000 7fbc0b40e640 1 -- 172.21.15.120:0/263725513 --> [v2:172.21.15.135:6810/1080459988,v1:172.21.15.135:6811/1080459988] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbc0406d9d0 con 0x7fbbe8074fb0 2024-02-06T22:18:06.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.071+0000 7fbbf9ffb640 1 -- 172.21.15.120:0/263725513 <== osd.6 v2:172.21.15.135:6810/1080459988 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fbc0406d9d0 con 0x7fbbe8074fb0 2024-02-06T22:18:06.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.071+0000 7fbbf9ffb640 10 client.16569.objecter ms_dispatch 0x7fbc04000b90 command_reply(tid 2: 0 ) v1 2024-02-06T22:18:06.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.071+0000 7fbbf9ffb640 10 client.16569.objecter _finish_command 2 = system:0 2024-02-06T22:18:06.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.071+0000 7fbbf9ffb640 20 client.16569.objecter put_session s=0x7fbbe80748e0 osd=6 4 2024-02-06T22:18:06.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.071+0000 7fbbf9ffb640 15 client.16569.objecter _session_command_op_remove 6 2 2024-02-06T22:18:06.072 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-02-06T22:18:06.072 INFO:teuthology.orchestra.run.smithi120.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-06T22:18:06.072 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-02-06T22:18:06.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.071+0000 7fbc0b40e640 10 librados: watch_flush enter 2024-02-06T22:18:06.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.071+0000 7fbc0b40e640 10 librados: watch_flush exit 2024-02-06T22:18:06.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.071+0000 7fbc0b40e640 10 client.16569.objecter close_session for osd.6 2024-02-06T22:18:06.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.071+0000 7fbc0b40e640 1 -- 172.21.15.120:0/263725513 >> [v2:172.21.15.135:6810/1080459988,v1:172.21.15.135:6811/1080459988] conn(0x7fbbe8074fb0 msgr2=0x7fbbe8077450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.071+0000 7fbc0b40e640 1 --2- 172.21.15.120:0/263725513 >> [v2:172.21.15.135:6810/1080459988,v1:172.21.15.135:6811/1080459988] conn(0x7fbbe8074fb0 0x7fbbe8077450 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.071+0000 7fbc0b40e640 20 client.16569.objecter put_session s=0x7fbbe80748e0 osd=6 1 2024-02-06T22:18:06.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.071+0000 7fbc0b40e640 20 client.16569.objecter shutdown clearing up homeless session... 2024-02-06T22:18:06.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.071+0000 7fbc0b40e640 10 client.16569.objecter successfully canceled tick 2024-02-06T22:18:06.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.072+0000 7fbc0b40e640 1 -- 172.21.15.120:0/263725513 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbbe806d220 msgr2=0x7fbbe806f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.072+0000 7fbc0b40e640 1 --2- 172.21.15.120:0/263725513 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fbbe806d220 0x7fbbe806f6e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fbc00009b40 tx=0x7fbc00009190 comp rx=0 tx=0).stop 2024-02-06T22:18:06.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.072+0000 7fbc0b40e640 10 monclient: shutdown 2024-02-06T22:18:06.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.072+0000 7fbc0b40e640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:06.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.072+0000 7fbc0b40e640 1 -- 172.21.15.120:0/263725513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc0413f8b0 msgr2=0x7fbc041db060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.072+0000 7fbc0b40e640 1 --2- 172.21.15.120:0/263725513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc0413f8b0 0x7fbc041db060 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009e80 tx=0x7fbbfc004a40 comp rx=0 tx=0).stop 2024-02-06T22:18:06.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.072+0000 7fbc099ab640 1 -- 172.21.15.120:0/263725513 reap_dead start 2024-02-06T22:18:06.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.072+0000 7fbc0b40e640 1 -- 172.21.15.120:0/263725513 shutdown_connections 2024-02-06T22:18:06.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.072+0000 7fbc0b40e640 1 -- 172.21.15.120:0/263725513 >> 172.21.15.120:0/263725513 conn(0x7fbc040b6160 msgr2=0x7fbc0413c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:06.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.072+0000 7fbc0b40e640 1 -- 172.21.15.120:0/263725513 shutdown_connections 2024-02-06T22:18:06.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.072+0000 7fbc0b40e640 1 -- 172.21.15.120:0/263725513 wait complete. 2024-02-06T22:18:06.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.072+0000 7fbc0b40e640 1 librados: shutdown 2024-02-06T22:18:06.083 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-02-06T22:18:06.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.183+0000 7f0d699c4640 10 monclient: get_monmap_and_config 2024-02-06T22:18:06.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.183+0000 7f0d699c4640 10 monclient: build_initial_monmap 2024-02-06T22:18:06.183 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.183+0000 7f0d699c4640 10 monclient: monmap: 2024-02-06T22:18:06.183 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:06.183 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:06.183 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:06.183347+0000 2024-02-06T22:18:06.184 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:06.183347+0000 2024-02-06T22:18:06.184 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:06.184 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.184 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:06.184 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:06.184 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:06.184 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:06.184 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:06.184 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:06.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.183+0000 7f0d699c4640 1 Processor -- start 2024-02-06T22:18:06.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.183+0000 7f0d699c4640 1 -- start start 2024-02-06T22:18:06.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.183+0000 7f0d699c4640 10 monclient: init 2024-02-06T22:18:06.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.184+0000 7f0d699c4640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:06.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.184+0000 7f0d699c4640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-02-06T22:18:06.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.184+0000 7f0d699c4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d640b54b0 addr v1:172.21.15.135:6789/0 2024-02-06T22:18:06.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.184+0000 7f0d699c4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6413ff40 0x7f0d640a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.184+0000 7f0d699c4640 10 monclient(hunting): picked mon.noname-b con 0x7f0d6413ff40 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:06.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.184+0000 7f0d699c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0d64142680 addr v1:172.21.15.120:6789/0 2024-02-06T22:18:06.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.184+0000 7f0d699c4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d64143200 con 0x7f0d64142680 2024-02-06T22:18:06.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.184+0000 7f0d699c4640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d640ab720 con 0x7f0d640b54b0 2024-02-06T22:18:06.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.184+0000 7f0d699c4640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:06.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.184+0000 7f0d699c4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d640ab520 con 0x7f0d6413ff40 2024-02-06T22:18:06.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.184+0000 7f0d699c4640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:06.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.184+0000 7f0d699c4640 10 monclient(hunting): authenticate will time out at 3516.520859s 2024-02-06T22:18:06.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.184+0000 7f0d637fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f0d64142680 0x7f0d64142af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47520/0 (socket says 172.21.15.120:47520) 2024-02-06T22:18:06.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.184+0000 7f0d637fe640 1 -- 172.21.15.120:0/1921600246 learned_addr learned my addr 172.21.15.120:0/1921600246 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T22:18:06.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.184+0000 7f0d627fc640 1 --2- 172.21.15.120:0/1921600246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6413ff40 0x7f0d640a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.185+0000 7f0d627fc640 10 monclient(hunting): get_auth_request con 0x7f0d6413ff40 auth_method 0 2024-02-06T22:18:06.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.185+0000 7f0d627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:06.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.185+0000 7f0d627fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:06.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.185+0000 7f0d627fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:06.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.185+0000 7f0d627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:06.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.185+0000 7f0d627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:06.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.185+0000 7f0d627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:06.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.187+0000 7f0d627fc640 10 monclient(hunting): handle_auth_done global_id 18743 payload 274 2024-02-06T22:18:06.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.187+0000 7f0d61ffb640 1 -- 172.21.15.120:0/1921600246 <== mon.0 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170166817 0 0) 0x7f0d64143200 con 0x7f0d64142680 2024-02-06T22:18:06.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.187+0000 7f0d627fc640 1 -- 172.21.15.120:0/1921600246 >> v1:172.21.15.135:6789/0 conn(0x7f0d640b54b0 legacy=0x7f0d6413f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.187+0000 7f0d627fc640 1 -- 172.21.15.120:0/1921600246 >> v1:172.21.15.120:6789/0 conn(0x7f0d64142680 legacy=0x7f0d64142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.187+0000 7f0d627fc640 10 monclient: _finish_hunting 0 2024-02-06T22:18:06.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.187+0000 7f0d627fc640 1 monclient: found mon.noname-b 2024-02-06T22:18:06.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.187+0000 7f0d627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:06.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.187+0000 7f0d627fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T22:18:06.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.187+0000 7f0d627fc640 1 -- 172.21.15.120:0/1921600246 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d640b4800 con 0x7f0d6413ff40 2024-02-06T22:18:06.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.187+0000 7f0d61ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-02-06T22:18:06.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.189+0000 7f0d627fc640 1 --2- 172.21.15.120:0/1921600246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6413ff40 0x7f0d640a9860 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7f0d580097d0 tx=0x7f0d5802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=417a2e4df30454a9 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:06.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d61ffb640 1 -- 172.21.15.120:0/1921600246 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58031890 con 0x7f0d6413ff40 2024-02-06T22:18:06.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:06.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d61ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:18:06.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d61ffb640 10 monclient: dump: 2024-02-06T22:18:06.194 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:06.194 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:06.194 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.194 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.194 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:06.195 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.195 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:06.195 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:06.195 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:06.195 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:06.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d61ffb640 10 monclient: _finish_auth 0 2024-02-06T22:18:06.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d61ffb640 10 monclient: _check_auth_tickets 2024-02-06T22:18:06.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:06.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d61ffb640 1 -- 172.21.15.120:0/1921600246 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d580319f0 con 0x7f0d6413ff40 2024-02-06T22:18:06.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d699c4640 5 monclient: authenticate success, global_id 18743 2024-02-06T22:18:06.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d699c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T22:18:06.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d61ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:06.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d699c4640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:06.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d61ffb640 1 -- 172.21.15.120:0/1921600246 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58031d80 con 0x7f0d6413ff40 2024-02-06T22:18:06.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d699c4640 4 set_mon_vals no callback set 2024-02-06T22:18:06.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:06.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d61ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T22:18:06.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d61ffb640 10 monclient: dump: 2024-02-06T22:18:06.196 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:06.196 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:06.196 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.196 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.196 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:06.196 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.196 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:06.196 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:06.196 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:06.197 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:06.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d699c4640 10 monclient: shutdown 2024-02-06T22:18:06.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d699c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:06.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d699c4640 1 -- 172.21.15.120:0/1921600246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6413ff40 msgr2=0x7f0d640a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d699c4640 1 --2- 172.21.15.120:0/1921600246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6413ff40 0x7f0d640a9860 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7f0d580097d0 tx=0x7f0d5802b930 comp rx=0 tx=0).stop 2024-02-06T22:18:06.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d699c4640 1 -- 172.21.15.120:0/1921600246 shutdown_connections 2024-02-06T22:18:06.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d699c4640 1 --2- 172.21.15.120:0/1921600246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6413ff40 0x7f0d640a9860 unknown :-1 s=CLOSED pgs=3351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d699c4640 1 -- 172.21.15.120:0/1921600246 >> 172.21.15.120:0/1921600246 conn(0x7f0d640b8660 msgr2=0x7f0d640b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:06.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d699c4640 1 -- 172.21.15.120:0/1921600246 shutdown_connections 2024-02-06T22:18:06.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.193+0000 7f0d699c4640 1 -- 172.21.15.120:0/1921600246 wait complete. 2024-02-06T22:18:06.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.194+0000 7f0d699c4640 10 monclient: build_initial_monmap 2024-02-06T22:18:06.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.194+0000 7f0d699c4640 10 monclient: monmap: 2024-02-06T22:18:06.201 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:06.201 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:06.201 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:06.201 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:06.201 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:06.201 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.201 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:06.201 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:06.201 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:06.201 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:06.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.194+0000 7f0d699c4640 1 librados: starting msgr at 2024-02-06T22:18:06.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.194+0000 7f0d699c4640 1 librados: starting objecter 2024-02-06T22:18:06.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.194+0000 7f0d699c4640 1 Processor -- start 2024-02-06T22:18:06.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 1 -- start start 2024-02-06T22:18:06.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 1 librados: setting wanted keys 2024-02-06T22:18:06.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 1 librados: calling monclient init 2024-02-06T22:18:06.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 10 monclient: init 2024-02-06T22:18:06.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:06.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T22:18:06.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d640b54b0 0x7f0d64137290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 10 monclient(hunting): picked mon.noname-c con 0x7f0d640b54b0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:06.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d6413ff40 0x7f0d64137c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 10 monclient(hunting): picked mon.noname-b con 0x7f0d6413ff40 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:06.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d64142680 0x7f0d641381b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 10 monclient(hunting): picked mon.noname-a con 0x7f0d64142680 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:06.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:06.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d641391c0 con 0x7f0d64142680 2024-02-06T22:18:06.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:06.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d640abce0 con 0x7f0d6413ff40 2024-02-06T22:18:06.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:06.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d64133dc0 con 0x7f0d640b54b0 2024-02-06T22:18:06.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:06.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.195+0000 7f0d699c4640 10 monclient(hunting): authenticate will time out at 3516.532262s 2024-02-06T22:18:06.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d637fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d64142680 0x7f0d641381b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d62ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d640b54b0 0x7f0d64137290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d627fc640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d6413ff40 0x7f0d64137c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d627fc640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d6413ff40 0x7f0d64137c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.120:51208/0 (socket says 172.21.15.120:51208) 2024-02-06T22:18:06.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d627fc640 1 -- 172.21.15.120:0/309865590 learned_addr learned my addr 172.21.15.120:0/309865590 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:06.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d62ffd640 10 monclient(hunting): get_auth_request con 0x7f0d640b54b0 auth_method 0 2024-02-06T22:18:06.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:06.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d62ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:06.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:06.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d637fe640 10 monclient(hunting): get_auth_request con 0x7f0d64142680 auth_method 0 2024-02-06T22:18:06.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:06.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d637fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:06.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d637fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:06.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d627fc640 10 monclient(hunting): get_auth_request con 0x7f0d6413ff40 auth_method 0 2024-02-06T22:18:06.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:06.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d627fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:06.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.196+0000 7f0d627fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:06.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.197+0000 7f0d627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:06.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.197+0000 7f0d627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:06.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.197+0000 7f0d627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:06.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.197+0000 7f0d62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:06.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.197+0000 7f0d62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:06.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.197+0000 7f0d62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:06.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.197+0000 7f0d637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:06.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.197+0000 7f0d637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:06.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.197+0000 7f0d637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:06.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.199+0000 7f0d627fc640 10 monclient(hunting): handle_auth_done global_id 21091 payload 824 2024-02-06T22:18:06.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.199+0000 7f0d627fc640 1 -- 172.21.15.120:0/309865590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d640b54b0 msgr2=0x7f0d64137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.199+0000 7f0d627fc640 1 --2- 172.21.15.120:0/309865590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d640b54b0 0x7f0d64137290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.199+0000 7f0d627fc640 1 -- 172.21.15.120:0/309865590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d64142680 msgr2=0x7f0d641381b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.199+0000 7f0d627fc640 1 --2- 172.21.15.120:0/309865590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d64142680 0x7f0d641381b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.199+0000 7f0d627fc640 10 monclient: _finish_hunting 0 2024-02-06T22:18:06.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.199+0000 7f0d627fc640 1 monclient: found mon.noname-b 2024-02-06T22:18:06.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.199+0000 7f0d627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:06.205 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.199+0000 7f0d627fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T22:18:06.206 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.199+0000 7f0d627fc640 1 -- 172.21.15.120:0/309865590 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d58009400 con 0x7f0d6413ff40 2024-02-06T22:18:06.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.202+0000 7f0d627fc640 1 --2- 172.21.15.120:0/309865590 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d6413ff40 0x7f0d64137c70 secure :-1 s=READY pgs=5101 cs=0 l=1 rev1=1 crypto rx=0x7f0d5802bf40 tx=0x7f0d580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:06.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.202+0000 7f0d4bfff640 10 client.?.objecter ms_handle_connect 0x7f0d6413ff40 2024-02-06T22:18:06.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.202+0000 7f0d4bfff640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:06.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.203+0000 7f0d4bfff640 1 -- 172.21.15.120:0/309865590 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58046450 con 0x7f0d6413ff40 2024-02-06T22:18:06.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.203+0000 7f0d4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:06.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.203+0000 7f0d4bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:18:06.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d4bfff640 10 monclient: dump: 2024-02-06T22:18:06.208 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:06.208 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:06.208 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.208 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.208 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:06.208 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.208 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:06.208 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:06.208 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:06.208 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:06.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d4bfff640 10 monclient: _finish_auth 0 2024-02-06T22:18:06.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d4bfff640 10 monclient: _check_auth_tickets 2024-02-06T22:18:06.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:06.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d699c4640 5 monclient: authenticate success, global_id 21091 2024-02-06T22:18:06.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d699c4640 10 monclient: _renew_subs 2024-02-06T22:18:06.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d699c4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:06.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d699c4640 1 -- 172.21.15.120:0/309865590 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d641393a0 con 0x7f0d6413ff40 2024-02-06T22:18:06.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d699c4640 10 client.21091.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:06.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d699c4640 10 monclient: _renew_subs 2024-02-06T22:18:06.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d699c4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:06.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d699c4640 1 -- 172.21.15.120:0/309865590 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d641db1e0 con 0x7f0d6413ff40 2024-02-06T22:18:06.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d699c4640 1 librados: init done 2024-02-06T22:18:06.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d699c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:06.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d699c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:06.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d699c4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:06.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.204+0000 7f0d699c4640 1 -- 172.21.15.120:0/309865590 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6414dee0 con 0x7f0d6413ff40 2024-02-06T22:18:06.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.205+0000 7f0d4bfff640 1 -- 172.21.15.120:0/309865590 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d580465b0 con 0x7f0d6413ff40 2024-02-06T22:18:06.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.205+0000 7f0d4bfff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:06.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.205+0000 7f0d4bfff640 1 -- 172.21.15.120:0/309865590 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58046940 con 0x7f0d6413ff40 2024-02-06T22:18:06.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.205+0000 7f0d4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:06.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.205+0000 7f0d4bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T22:18:06.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.205+0000 7f0d4bfff640 10 monclient: dump: 2024-02-06T22:18:06.210 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:06.210 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:06.210 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.210 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.210 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:06.210 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.210 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:06.210 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:06.210 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:06.210 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:06.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.205+0000 7f0d617fa640 4 set_mon_vals no callback set 2024-02-06T22:18:06.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.448+0000 7f0d4bfff640 1 -- 172.21.15.120:0/309865590 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0d5804f3f0 con 0x7f0d6413ff40 2024-02-06T22:18:06.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.449+0000 7f0d4bfff640 1 --2- 172.21.15.120:0/309865590 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f0d5006d1a0 0x7f0d5006f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.449+0000 7f0d62ffd640 1 --2- 172.21.15.120:0/309865590 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f0d5006d1a0 0x7f0d5006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.449+0000 7f0d62ffd640 10 monclient: get_auth_request con 0x7f0d5006d1a0 auth_method 0 2024-02-06T22:18:06.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.450+0000 7f0d4bfff640 10 client.21091.objecter ms_handle_connect 0x7f0d5006d1a0 2024-02-06T22:18:06.450 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.450+0000 7f0d62ffd640 1 --2- 172.21.15.120:0/309865590 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f0d5006d1a0 0x7f0d5006f660 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c001780 tx=0x7f0d5c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:06.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.452+0000 7f0d4bfff640 1 -- 172.21.15.120:0/309865590 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f0d5807b820 con 0x7f0d6413ff40 2024-02-06T22:18:06.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.452+0000 7f0d4bfff640 10 client.21091.objecter ms_dispatch 0x7f0d64000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:06.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.452+0000 7f0d4bfff640 3 client.21091.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:06.452 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.452+0000 7f0d4bfff640 3 client.21091.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:06.453 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.452+0000 7f0d4bfff640 20 client.21091.objecter dump_active .. 0 homeless 2024-02-06T22:18:06.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.463+0000 7f0d4bfff640 1 -- 172.21.15.120:0/309865590 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0d58077ac0 con 0x7f0d6413ff40 2024-02-06T22:18:06.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.463+0000 7f0d4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:06.463 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.463+0000 7f0d4bfff640 10 monclient: _finish_command 1 = system:0 2024-02-06T22:18:06.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.646+0000 7f0d699c4640 1 -- 172.21.15.120:0/309865590 --> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d640aa450 con 0x7f0d5006d1a0 2024-02-06T22:18:06.647 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.646+0000 7f0d4bfff640 1 -- 172.21.15.120:0/309865590 <== mgr.4100 v2:172.21.15.135:6800/315664144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0d5c001420 con 0x7f0d5006d1a0 2024-02-06T22:18:06.647 INFO:teuthology.orchestra.run.smithi120.stderr:instructed osd(s) 6 to deep-scrub 2024-02-06T22:18:06.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 10 librados: watch_flush enter 2024-02-06T22:18:06.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 10 librados: watch_flush exit 2024-02-06T22:18:06.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 20 client.21091.objecter shutdown clearing up homeless session... 2024-02-06T22:18:06.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 10 client.21091.objecter successfully canceled tick 2024-02-06T22:18:06.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 1 -- 172.21.15.120:0/309865590 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f0d5006d1a0 msgr2=0x7f0d5006f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 1 --2- 172.21.15.120:0/309865590 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f0d5006d1a0 0x7f0d5006f660 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c001780 tx=0x7f0d5c009040 comp rx=0 tx=0).stop 2024-02-06T22:18:06.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 10 monclient: shutdown 2024-02-06T22:18:06.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:06.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 1 -- 172.21.15.120:0/309865590 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d6413ff40 msgr2=0x7f0d64137c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 1 --2- 172.21.15.120:0/309865590 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d6413ff40 0x7f0d64137c70 secure :-1 s=READY pgs=5101 cs=0 l=1 rev1=1 crypto rx=0x7f0d5802bf40 tx=0x7f0d580046c0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 1 -- 172.21.15.120:0/309865590 shutdown_connections 2024-02-06T22:18:06.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 1 --2- 172.21.15.120:0/309865590 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f0d5006d1a0 0x7f0d5006f660 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 1 --2- 172.21.15.120:0/309865590 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d6413ff40 0x7f0d64137c70 unknown :-1 s=CLOSED pgs=5101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 1 --2- 172.21.15.120:0/309865590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d64142680 0x7f0d641381b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 1 --2- 172.21.15.120:0/309865590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d640b54b0 0x7f0d64137290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.648+0000 7f0d699c4640 1 -- 172.21.15.120:0/309865590 >> 172.21.15.120:0/309865590 conn(0x7f0d640b8660 msgr2=0x7f0d6413c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:06.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.649+0000 7f0d699c4640 1 -- 172.21.15.120:0/309865590 shutdown_connections 2024-02-06T22:18:06.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.649+0000 7f0d699c4640 1 -- 172.21.15.120:0/309865590 wait complete. 2024-02-06T22:18:06.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.649+0000 7f0d699c4640 1 librados: shutdown 2024-02-06T22:18:06.660 INFO:tasks.ceph:Scrubbing osd.7 2024-02-06T22:18:06.661 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-02-06T22:18:06.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.767+0000 7fdc0d379640 10 monclient: get_monmap_and_config 2024-02-06T22:18:06.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.767+0000 7fdc0d379640 10 monclient: build_initial_monmap 2024-02-06T22:18:06.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.767+0000 7fdc0d379640 10 monclient: monmap: 2024-02-06T22:18:06.768 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:06.768 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:06.768 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:06.768062+0000 2024-02-06T22:18:06.768 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:06.768062+0000 2024-02-06T22:18:06.768 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:06.769 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.769 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:06.769 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:06.769 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:06.769 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:06.769 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:06.769 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:06.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.768+0000 7fdc0d379640 1 Processor -- start 2024-02-06T22:18:06.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.768+0000 7fdc0d379640 1 -- start start 2024-02-06T22:18:06.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.768+0000 7fdc0d379640 10 monclient: init 2024-02-06T22:18:06.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:06.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-02-06T22:18:06.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdc080b5240 0x7fdc08142fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 10 monclient(hunting): picked mon.noname-c con 0x7fdc080b5240 addr v2:172.21.15.135:3300/0 2024-02-06T22:18:06.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdc080b5620 0x7fdc080afa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 10 monclient(hunting): picked mon.noname-a con 0x7fdc080b5620 addr v2:172.21.15.120:3300/0 2024-02-06T22:18:06.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc080aef10 0x7fdc080affa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 10 monclient(hunting): picked mon.noname-b con 0x7fdc080aef10 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:06.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:06.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc080b2240 con 0x7fdc080b5620 2024-02-06T22:18:06.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:06.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc080b13a0 con 0x7fdc080b5240 2024-02-06T22:18:06.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:06.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc08142760 con 0x7fdc080aef10 2024-02-06T22:18:06.772 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:06.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc0d379640 10 monclient(hunting): authenticate will time out at 3517.105654s 2024-02-06T22:18:06.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc067fc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdc080b5620 0x7fdc080afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc077fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc080aef10 0x7fdc080affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc06ffd640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdc080b5240 0x7fdc08142fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.769+0000 7fdc067fc640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdc080b5620 0x7fdc080afa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:49682/0 (socket says 172.21.15.120:49682) 2024-02-06T22:18:06.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc067fc640 1 -- 172.21.15.120:0/2348380834 learned_addr learned my addr 172.21.15.120:0/2348380834 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:06.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc067fc640 10 monclient(hunting): get_auth_request con 0x7fdc080b5620 auth_method 0 2024-02-06T22:18:06.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:06.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc067fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:06.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc067fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:06.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc077fe640 10 monclient(hunting): get_auth_request con 0x7fdc080aef10 auth_method 0 2024-02-06T22:18:06.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:06.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc077fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:06.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc077fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:06.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc06ffd640 10 monclient(hunting): get_auth_request con 0x7fdc080b5240 auth_method 0 2024-02-06T22:18:06.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:06.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc06ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:06.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:06.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:06.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:06.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:06.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:06.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:06.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:06.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:06.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:06.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.770+0000 7fdc06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:06.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.772+0000 7fdc067fc640 10 monclient(hunting): handle_auth_done global_id 16578 payload 274 2024-02-06T22:18:06.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.772+0000 7fdc067fc640 1 -- 172.21.15.120:0/2348380834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc080aef10 msgr2=0x7fdc080affa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:06.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.772+0000 7fdc067fc640 1 --2- 172.21.15.120:0/2348380834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc080aef10 0x7fdc080affa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.772+0000 7fdc067fc640 1 -- 172.21.15.120:0/2348380834 >> v2:172.21.15.135:3300/0 conn(0x7fdc080b5240 msgr2=0x7fdc08142fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:06.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.772+0000 7fdc067fc640 1 --2- 172.21.15.120:0/2348380834 >> v2:172.21.15.135:3300/0 conn(0x7fdc080b5240 0x7fdc08142fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.772+0000 7fdc067fc640 10 monclient: _finish_hunting 0 2024-02-06T22:18:06.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.772+0000 7fdc067fc640 1 monclient: found mon.noname-a 2024-02-06T22:18:06.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.772+0000 7fdc067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:06.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.772+0000 7fdc067fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:18:06.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.772+0000 7fdc067fc640 1 -- 172.21.15.120:0/2348380834 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc080b15a0 con 0x7fdc080b5620 2024-02-06T22:18:06.776 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.774+0000 7fdc067fc640 1 --2- 172.21.15.120:0/2348380834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc080b5620 0x7fdc080afa60 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7fdc0806de20 tx=0x7fdbfc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=147dfa06635bf00d server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:06.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc05ffb640 1 -- 172.21.15.120:0/2348380834 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc0317e0 con 0x7fdc080b5620 2024-02-06T22:18:06.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:06.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc05ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:18:06.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc05ffb640 10 monclient: dump: 2024-02-06T22:18:06.778 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:06.778 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:06.778 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.778 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.778 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:06.778 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.778 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:06.778 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:06.778 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc05ffb640 10 monclient: _finish_auth 0 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc05ffb640 10 monclient: _check_auth_tickets 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc05ffb640 1 -- 172.21.15.120:0/2348380834 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbfc031940 con 0x7fdc080b5620 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc05ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc05ffb640 1 -- 172.21.15.120:0/2348380834 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc031cd0 con 0x7fdc080b5620 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc0d379640 5 monclient: authenticate success, global_id 16578 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc0d379640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc0d379640 4 set_mon_vals no callback set 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc05ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.777+0000 7fdc05ffb640 10 monclient: dump: 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:06.779 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.780 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.780 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:06.780 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.780 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:06.780 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:06.780 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:06.780 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:06.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.778+0000 7fdc0d379640 10 monclient: shutdown 2024-02-06T22:18:06.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.778+0000 7fdc0d379640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:06.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.778+0000 7fdc0d379640 1 -- 172.21.15.120:0/2348380834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc080b5620 msgr2=0x7fdc080afa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.778+0000 7fdc0d379640 1 --2- 172.21.15.120:0/2348380834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc080b5620 0x7fdc080afa60 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7fdc0806de20 tx=0x7fdbfc02b880 comp rx=0 tx=0).stop 2024-02-06T22:18:06.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.778+0000 7fdc0d379640 1 -- 172.21.15.120:0/2348380834 shutdown_connections 2024-02-06T22:18:06.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.778+0000 7fdc0d379640 1 --2- 172.21.15.120:0/2348380834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc080aef10 0x7fdc080affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.778+0000 7fdc0d379640 1 --2- 172.21.15.120:0/2348380834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc080b5620 0x7fdc080afa60 unknown :-1 s=CLOSED pgs=3359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.778+0000 7fdc0d379640 1 --2- 172.21.15.120:0/2348380834 >> v2:172.21.15.135:3300/0 conn(0x7fdc080b5240 0x7fdc08142fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.778+0000 7fdc0d379640 1 -- 172.21.15.120:0/2348380834 >> 172.21.15.120:0/2348380834 conn(0x7fdc080b8660 msgr2=0x7fdc080b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:06.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.778+0000 7fdc0d379640 1 -- 172.21.15.120:0/2348380834 shutdown_connections 2024-02-06T22:18:06.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.778+0000 7fdc0d379640 1 -- 172.21.15.120:0/2348380834 wait complete. 2024-02-06T22:18:06.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.779+0000 7fdc0d379640 10 monclient: build_initial_monmap 2024-02-06T22:18:06.785 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.779+0000 7fdc0d379640 10 monclient: monmap: 2024-02-06T22:18:06.785 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:06.785 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:06.785 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:06.785 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:06.785 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:06.785 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.785 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:06.785 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:06.785 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:06.785 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:06.785 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.779+0000 7fdc0d379640 1 librados: starting msgr at 2024-02-06T22:18:06.785 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.779+0000 7fdc0d379640 1 librados: starting objecter 2024-02-06T22:18:06.785 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.779+0000 7fdc0d379640 1 Processor -- start 2024-02-06T22:18:06.785 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.779+0000 7fdc0d379640 1 -- start start 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.779+0000 7fdc0d379640 1 librados: setting wanted keys 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.779+0000 7fdc0d379640 1 librados: calling monclient init 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.779+0000 7fdc0d379640 10 monclient: init 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 10 monclient: _add_conns ranks=[0,1,2] 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc080aef10 0x7fdc081dad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 10 monclient(hunting): picked mon.noname-a con 0x7fdc080aef10 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc080b5240 0x7fdc080ad290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 10 monclient(hunting): picked mon.noname-b con 0x7fdc080b5240 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc0813dd00 0x7fdc080ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 10 monclient(hunting): picked mon.noname-c con 0x7fdc0813dd00 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc080ace00 con 0x7fdc080aef10 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc080add10 con 0x7fdc080b5240 2024-02-06T22:18:06.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:06.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc080ade80 con 0x7fdc0813dd00 2024-02-06T22:18:06.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:06.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc0d379640 10 monclient(hunting): authenticate will time out at 3517.116872s 2024-02-06T22:18:06.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc06ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc080aef10 0x7fdc081dad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc077fe640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc0813dd00 0x7fdc080ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.780+0000 7fdc067fc640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc080b5240 0x7fdc080ad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc06ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc080aef10 0x7fdc081dad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:49692/0 (socket says 172.21.15.120:49692) 2024-02-06T22:18:06.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc06ffd640 1 -- 172.21.15.120:0/429497160 learned_addr learned my addr 172.21.15.120:0/429497160 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:06.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc06ffd640 10 monclient(hunting): get_auth_request con 0x7fdc080aef10 auth_method 0 2024-02-06T22:18:06.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:06.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc06ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:06.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:06.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc067fc640 10 monclient(hunting): get_auth_request con 0x7fdc080b5240 auth_method 0 2024-02-06T22:18:06.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:06.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc067fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:06.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc067fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:06.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc077fe640 10 monclient(hunting): get_auth_request con 0x7fdc0813dd00 auth_method 0 2024-02-06T22:18:06.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:06.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc077fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:06.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc077fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:06.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:06.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:06.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:06.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:06.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:06.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:06.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:06.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:06.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.781+0000 7fdc077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:06.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.784+0000 7fdc06ffd640 10 monclient(hunting): handle_auth_done global_id 16581 payload 824 2024-02-06T22:18:06.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.784+0000 7fdc06ffd640 1 -- 172.21.15.120:0/429497160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc0813dd00 msgr2=0x7fdc080ad7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.784+0000 7fdc06ffd640 1 --2- 172.21.15.120:0/429497160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc0813dd00 0x7fdc080ad7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.784+0000 7fdc06ffd640 1 -- 172.21.15.120:0/429497160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc080b5240 msgr2=0x7fdc080ad290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.784+0000 7fdc06ffd640 1 --2- 172.21.15.120:0/429497160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc080b5240 0x7fdc080ad290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.784+0000 7fdc06ffd640 10 monclient: _finish_hunting 0 2024-02-06T22:18:06.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.784+0000 7fdc06ffd640 1 monclient: found mon.noname-a 2024-02-06T22:18:06.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.784+0000 7fdc06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:06.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.784+0000 7fdc06ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:18:06.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.784+0000 7fdc06ffd640 1 -- 172.21.15.120:0/429497160 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc0094d0 con 0x7fdc080aef10 2024-02-06T22:18:06.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.786+0000 7fdc06ffd640 1 --2- 172.21.15.120:0/429497160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc080aef10 0x7fdc081dad90 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800e9e0 tx=0x7fdbf800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:06.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.786+0000 7fdbef7fe640 10 client.?.objecter ms_handle_connect 0x7fdc080aef10 2024-02-06T22:18:06.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.786+0000 7fdbef7fe640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:06.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdbef7fe640 1 -- 172.21.15.120:0/429497160 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8009670 con 0x7fdc080aef10 2024-02-06T22:18:06.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdbef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:06.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdbef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:18:06.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdbef7fe640 10 monclient: dump: 2024-02-06T22:18:06.792 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:06.792 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:06.792 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.792 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.792 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:06.792 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.792 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:06.792 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdbef7fe640 10 monclient: _finish_auth 0 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdbef7fe640 10 monclient: _check_auth_tickets 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdbef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdbef7fe640 1 -- 172.21.15.120:0/429497160 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf800c9d0 con 0x7fdc080aef10 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdbef7fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdbef7fe640 1 -- 172.21.15.120:0/429497160 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf800cd80 con 0x7fdc080aef10 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdc0d379640 5 monclient: authenticate success, global_id 16581 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdc0d379640 10 monclient: _renew_subs 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdc0d379640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdc0d379640 1 -- 172.21.15.120:0/429497160 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc080ac950 con 0x7fdc080aef10 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdbef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdbef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdc057fa640 4 set_mon_vals no callback set 2024-02-06T22:18:06.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdbef7fe640 10 monclient: dump: 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdc0d379640 10 client.16581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdc0d379640 10 monclient: _renew_subs 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdc0d379640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdc0d379640 1 -- 172.21.15.120:0/429497160 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc081db990 con 0x7fdc080aef10 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.790+0000 7fdc0d379640 1 librados: init done 2024-02-06T22:18:06.794 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.791+0000 7fdc0d379640 10 client.16581.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:06.795 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.791+0000 7fdc0d379640 15 client.16581.objecter _session_command_op_assign -1 1 2024-02-06T22:18:06.795 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.791+0000 7fdc0d379640 20 client.16581.objecter _get_session osd=-1 returning homeless 2024-02-06T22:18:06.795 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.791+0000 7fdc0d379640 10 client.16581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:06.795 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.791+0000 7fdc0d379640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:06.795 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.791+0000 7fdc0d379640 1 -- 172.21.15.120:0/429497160 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdc080b04e0 con 0x7fdc080aef10 2024-02-06T22:18:06.812 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.812+0000 7fdbef7fe640 1 -- 172.21.15.120:0/429497160 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdbf8013470 con 0x7fdc080aef10 2024-02-06T22:18:06.812 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.812+0000 7fdbef7fe640 1 --2- 172.21.15.120:0/429497160 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fdbf406d1f0 0x7fdbf406f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.813 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.812+0000 7fdc067fc640 1 --2- 172.21.15.120:0/429497160 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fdbf406d1f0 0x7fdbf406f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.813 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.812+0000 7fdc067fc640 10 monclient: get_auth_request con 0x7fdbf406d1f0 auth_method 0 2024-02-06T22:18:06.813 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.813+0000 7fdc067fc640 1 --2- 172.21.15.120:0/429497160 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fdbf406d1f0 0x7fdbf406f6b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc036000 tx=0x7fdbfc03d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:06.813 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.813+0000 7fdbef7fe640 10 client.16581.objecter ms_handle_connect 0x7fdbf406d1f0 2024-02-06T22:18:06.816 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 1 -- 172.21.15.120:0/429497160 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fdbf8058f80 con 0x7fdc080aef10 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 10 client.16581.objecter ms_dispatch 0x7fdc08000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 3 client.16581.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 3 client.16581.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 10 client.16581.objecter checking command 1 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 1 --2- 172.21.15.120:0/429497160 >> [v2:172.21.15.135:6825/3740758337,v1:172.21.15.135:6827/3740758337] conn(0x7fdbf4074f80 0x7fdbf4077420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 20 client.16581.objecter _get_session s=0x7fdbf40748b0 osd=7 3 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 20 client.16581.objecter put_session s=0x7fdbf40748b0 osd=7 3 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 15 client.16581.objecter _session_command_op_remove -1 1 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 20 client.16581.objecter _get_session s=0x7fdbf40748b0 osd=7 3 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 20 client.16581.objecter get_session s=0x7fdbf40748b0 osd=7 3 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 15 client.16581.objecter _session_command_op_assign 7 1 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 20 client.16581.objecter put_session s=0x7fdbf40748b0 osd=7 4 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 10 client.16581.objecter _send_command 1 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 1 -- 172.21.15.120:0/429497160 --> [v2:172.21.15.135:6825/3740758337,v1:172.21.15.135:6827/3740758337] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdbf4077af0 con 0x7fdbf4074f80 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 20 client.16581.objecter dump_active .. 0 homeless 2024-02-06T22:18:06.817 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 1 -- 172.21.15.120:0/429497160 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_get_version_reply(handle=1 version=1346) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbf8055220 con 0x7fdc080aef10 2024-02-06T22:18:06.818 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.815+0000 7fdbef7fe640 10 monclient: handle_get_version_reply finishing 1 version 1346 2024-02-06T22:18:06.818 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.816+0000 7fdc077fe640 1 --2- 172.21.15.120:0/429497160 >> [v2:172.21.15.135:6825/3740758337,v1:172.21.15.135:6827/3740758337] conn(0x7fdbf4074f80 0x7fdbf4077420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.818 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.817+0000 7fdc077fe640 10 monclient: get_auth_request con 0x7fdbf4074f80 auth_method 0 2024-02-06T22:18:06.819 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.819+0000 7fdc077fe640 1 --2- 172.21.15.120:0/429497160 >> [v2:172.21.15.135:6825/3740758337,v1:172.21.15.135:6827/3740758337] conn(0x7fdbf4074f80 0x7fdbf4077420 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:06.820 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.819+0000 7fdbef7fe640 10 client.16581.objecter ms_handle_connect 0x7fdbf4074f80 2024-02-06T22:18:06.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.842+0000 7fdbef7fe640 1 -- 172.21.15.120:0/429497160 <== osd.7 v2:172.21.15.135:6825/3740758337 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdbf4077af0 con 0x7fdbf4074f80 2024-02-06T22:18:06.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.842+0000 7fdbef7fe640 10 client.16581.objecter ms_dispatch 0x7fdc08000b90 command_reply(tid 1: 0 ) v1 2024-02-06T22:18:06.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.842+0000 7fdbef7fe640 10 client.16581.objecter _finish_command 1 = system:0 2024-02-06T22:18:06.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.842+0000 7fdbef7fe640 20 client.16581.objecter put_session s=0x7fdbf40748b0 osd=7 4 2024-02-06T22:18:06.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.842+0000 7fdbef7fe640 15 client.16581.objecter _session_command_op_remove 7 1 2024-02-06T22:18:06.857 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.857+0000 7fdc0d379640 10 client.16581.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-02-06T22:18:06.858 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.857+0000 7fdc0d379640 15 client.16581.objecter _session_command_op_assign -1 2 2024-02-06T22:18:06.858 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.857+0000 7fdc0d379640 20 client.16581.objecter _get_session s=0x7fdbf40748b0 osd=7 3 2024-02-06T22:18:06.858 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.857+0000 7fdc0d379640 20 client.16581.objecter put_session s=0x7fdbf40748b0 osd=7 3 2024-02-06T22:18:06.858 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.857+0000 7fdc0d379640 20 client.16581.objecter _get_session s=0x7fdbf40748b0 osd=7 3 2024-02-06T22:18:06.858 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.857+0000 7fdc0d379640 15 client.16581.objecter _session_command_op_remove -1 2 2024-02-06T22:18:06.858 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.857+0000 7fdc0d379640 20 client.16581.objecter get_session s=0x7fdbf40748b0 osd=7 3 2024-02-06T22:18:06.858 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.857+0000 7fdc0d379640 15 client.16581.objecter _session_command_op_assign 7 2 2024-02-06T22:18:06.858 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.857+0000 7fdc0d379640 20 client.16581.objecter put_session s=0x7fdbf40748b0 osd=7 4 2024-02-06T22:18:06.858 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.857+0000 7fdc0d379640 10 client.16581.objecter _send_command 2 2024-02-06T22:18:06.858 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.857+0000 7fdc0d379640 1 -- 172.21.15.120:0/429497160 --> [v2:172.21.15.135:6825/3740758337,v1:172.21.15.135:6827/3740758337] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdc080b2430 con 0x7fdbf4074f80 2024-02-06T22:18:06.870 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.870+0000 7fdbef7fe640 1 -- 172.21.15.120:0/429497160 <== osd.7 v2:172.21.15.135:6825/3740758337 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fdc00011470 con 0x7fdbf4074f80 2024-02-06T22:18:06.871 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.870+0000 7fdbef7fe640 10 client.16581.objecter ms_dispatch 0x7fdc08000b90 command_reply(tid 2: 0 ) v1 2024-02-06T22:18:06.871 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.870+0000 7fdbef7fe640 10 client.16581.objecter _finish_command 2 = system:0 2024-02-06T22:18:06.871 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.870+0000 7fdbef7fe640 20 client.16581.objecter put_session s=0x7fdbf40748b0 osd=7 4 2024-02-06T22:18:06.871 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.870+0000 7fdbef7fe640 15 client.16581.objecter _session_command_op_remove 7 2 2024-02-06T22:18:06.871 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-02-06T22:18:06.871 INFO:teuthology.orchestra.run.smithi120.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-06T22:18:06.871 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-02-06T22:18:06.873 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.870+0000 7fdc0d379640 10 librados: watch_flush enter 2024-02-06T22:18:06.873 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.870+0000 7fdc0d379640 10 librados: watch_flush exit 2024-02-06T22:18:06.873 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.870+0000 7fdc0d379640 10 client.16581.objecter close_session for osd.7 2024-02-06T22:18:06.873 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.870+0000 7fdc0d379640 1 -- 172.21.15.120:0/429497160 >> [v2:172.21.15.135:6825/3740758337,v1:172.21.15.135:6827/3740758337] conn(0x7fdbf4074f80 msgr2=0x7fdbf4077420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.873 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.870+0000 7fdc0d379640 1 --2- 172.21.15.120:0/429497160 >> [v2:172.21.15.135:6825/3740758337,v1:172.21.15.135:6827/3740758337] conn(0x7fdbf4074f80 0x7fdbf4077420 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.873 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.870+0000 7fdc0d379640 20 client.16581.objecter put_session s=0x7fdbf40748b0 osd=7 1 2024-02-06T22:18:06.873 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.870+0000 7fdc0d379640 20 client.16581.objecter shutdown clearing up homeless session... 2024-02-06T22:18:06.873 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.870+0000 7fdc0d379640 10 client.16581.objecter successfully canceled tick 2024-02-06T22:18:06.873 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.871+0000 7fdc0d379640 1 -- 172.21.15.120:0/429497160 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fdbf406d1f0 msgr2=0x7fdbf406f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.873 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.871+0000 7fdc0d379640 1 --2- 172.21.15.120:0/429497160 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7fdbf406d1f0 0x7fdbf406f6b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc036000 tx=0x7fdbfc03d040 comp rx=0 tx=0).stop 2024-02-06T22:18:06.873 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.871+0000 7fdc0d379640 10 monclient: shutdown 2024-02-06T22:18:06.873 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.871+0000 7fdc0d379640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:06.873 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.871+0000 7fdc0d379640 1 -- 172.21.15.120:0/429497160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc080aef10 msgr2=0x7fdc081dad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.874 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.871+0000 7fdc0d379640 1 --2- 172.21.15.120:0/429497160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc080aef10 0x7fdc081dad90 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800e9e0 tx=0x7fdbf800eea0 comp rx=0 tx=0).stop 2024-02-06T22:18:06.874 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.871+0000 7fdc077fe640 1 -- 172.21.15.120:0/429497160 reap_dead start 2024-02-06T22:18:06.874 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.871+0000 7fdc0d379640 1 -- 172.21.15.120:0/429497160 shutdown_connections 2024-02-06T22:18:06.874 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.871+0000 7fdc0d379640 1 -- 172.21.15.120:0/429497160 >> 172.21.15.120:0/429497160 conn(0x7fdc080b8660 msgr2=0x7fdc080b47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:06.874 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.871+0000 7fdc0d379640 1 -- 172.21.15.120:0/429497160 shutdown_connections 2024-02-06T22:18:06.874 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.871+0000 7fdc0d379640 1 -- 172.21.15.120:0/429497160 wait complete. 2024-02-06T22:18:06.874 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.871+0000 7fdc0d379640 1 librados: shutdown 2024-02-06T22:18:06.882 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-02-06T22:18:06.982 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.981+0000 7f4d6907b640 10 monclient: get_monmap_and_config 2024-02-06T22:18:06.982 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.981+0000 7f4d6907b640 10 monclient: build_initial_monmap 2024-02-06T22:18:06.982 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.981+0000 7f4d6907b640 10 monclient: monmap: 2024-02-06T22:18:06.982 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:06.982 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:06.982 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:06.981902+0000 2024-02-06T22:18:06.982 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:06.981902+0000 2024-02-06T22:18:06.982 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:06.982 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.982 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:06.982 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:06.982 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:06.983 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:06.983 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:06.983 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:06.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.982+0000 7f4d6907b640 1 Processor -- start 2024-02-06T22:18:06.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.982+0000 7f4d6907b640 1 -- start start 2024-02-06T22:18:06.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.982+0000 7f4d6907b640 10 monclient: init 2024-02-06T22:18:06.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.982+0000 7f4d6907b640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:06.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.982+0000 7f4d6907b640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-02-06T22:18:06.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.982+0000 7f4d6907b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d640ae8e0 addr v1:172.21.15.135:6789/0 2024-02-06T22:18:06.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.982+0000 7f4d6907b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d640aecc0 addr v1:172.21.15.120:6789/0 2024-02-06T22:18:06.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.982+0000 7f4d6907b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4d64072540 0x7f4d640729b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:06.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.982+0000 7f4d6907b640 10 monclient(hunting): picked mon.noname-a con 0x7f4d64072540 addr v2:172.21.15.120:3300/0 2024-02-06T22:18:06.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.982+0000 7f4d6907b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d640718d0 con 0x7f4d640aecc0 2024-02-06T22:18:06.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.982+0000 7f4d6907b640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d64071b20 con 0x7f4d640ae8e0 2024-02-06T22:18:06.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.982+0000 7f4d6907b640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:06.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.982+0000 7f4d6907b640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d6406ff30 con 0x7f4d64072540 2024-02-06T22:18:06.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.982+0000 7f4d6907b640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:06.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.982+0000 7f4d6907b640 10 monclient(hunting): authenticate will time out at 3517.319505s 2024-02-06T22:18:06.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.983+0000 7f4d6359e640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4d64072540 0x7f4d640729b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:06.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.983+0000 7f4d62d9d640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f4d640ae8e0 0x7f4d641431f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.120:40122/0 (socket says 172.21.15.120:40122) 2024-02-06T22:18:06.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.983+0000 7f4d62d9d640 1 -- 172.21.15.120:0/3910515716 learned_addr learned my addr 172.21.15.120:0/3910515716 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T22:18:06.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.984+0000 7f4d6359e640 10 monclient(hunting): get_auth_request con 0x7f4d64072540 auth_method 0 2024-02-06T22:18:06.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.984+0000 7f4d6359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:06.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.984+0000 7f4d6359e640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:06.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.984+0000 7f4d6359e640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:06.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.985+0000 7f4d6359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:06.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.985+0000 7f4d6359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:06.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.985+0000 7f4d6359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:06.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.985+0000 7f4d6259c640 1 -- 172.21.15.120:0/3910515716 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604458320 0 0) 0x7f4d64071b20 con 0x7f4d640ae8e0 2024-02-06T22:18:06.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.985+0000 7f4d6259c640 10 monclient(hunting): _negotiate 2024-02-06T22:18:06.987 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.985+0000 7f4d6259c640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:06.987 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.985+0000 7f4d6259c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:06.987 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.985+0000 7f4d6259c640 10 monclient(hunting): my global_id is 21100 2024-02-06T22:18:06.987 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.985+0000 7f4d6259c640 1 -- 172.21.15.120:0/3910515716 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d50003520 con 0x7f4d640ae8e0 2024-02-06T22:18:06.987 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.986+0000 7f4d6359e640 10 monclient(hunting): handle_auth_done global_id 16584 payload 274 2024-02-06T22:18:06.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.986+0000 7f4d6359e640 1 -- 172.21.15.120:0/3910515716 >> v1:172.21.15.135:6789/0 conn(0x7f4d640ae8e0 legacy=0x7f4d641431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.986+0000 7f4d6359e640 1 -- 172.21.15.120:0/3910515716 >> v1:172.21.15.120:6789/0 conn(0x7f4d640aecc0 legacy=0x7f4d6406f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:06.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.986+0000 7f4d6359e640 10 monclient: _finish_hunting 0 2024-02-06T22:18:06.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.986+0000 7f4d6359e640 1 monclient: found mon.noname-a 2024-02-06T22:18:06.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.986+0000 7f4d6359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:06.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.986+0000 7f4d6359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:18:06.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.986+0000 7f4d6359e640 1 -- 172.21.15.120:0/3910515716 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d64142890 con 0x7f4d64072540 2024-02-06T22:18:06.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.990+0000 7f4d6359e640 1 --2- 172.21.15.120:0/3910515716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d64072540 0x7f4d640729b0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c009d90 tx=0x7f4d5c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e467ea5732fff1b server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:06.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6259c640 1 -- 172.21.15.120:0/3910515716 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c02cc60 con 0x7f4d64072540 2024-02-06T22:18:06.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:06.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:18:06.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6259c640 10 monclient: dump: 2024-02-06T22:18:06.995 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:06.995 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:06.995 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.995 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.995 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:06.995 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.995 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:06.996 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:06.996 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:06.996 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:06.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6259c640 10 monclient: _finish_auth 0 2024-02-06T22:18:06.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6259c640 10 monclient: _check_auth_tickets 2024-02-06T22:18:06.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:06.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6259c640 1 -- 172.21.15.120:0/3910515716 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5c02cdc0 con 0x7f4d64072540 2024-02-06T22:18:06.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6259c640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:06.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6259c640 1 -- 172.21.15.120:0/3910515716 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c034660 con 0x7f4d64072540 2024-02-06T22:18:06.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6907b640 5 monclient: authenticate success, global_id 16584 2024-02-06T22:18:06.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6907b640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:06.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6907b640 4 set_mon_vals no callback set 2024-02-06T22:18:06.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:06.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:18:06.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.994+0000 7f4d6259c640 10 monclient: dump: 2024-02-06T22:18:06.997 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:06.997 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:06.997 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.997 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:06.997 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:06.997 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:06.997 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:06.997 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:06.997 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:06.997 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:07.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.995+0000 7f4d6907b640 10 monclient: shutdown 2024-02-06T22:18:07.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.995+0000 7f4d6907b640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:07.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.995+0000 7f4d6907b640 1 -- 172.21.15.120:0/3910515716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d64072540 msgr2=0x7f4d640729b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:07.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.995+0000 7f4d6907b640 1 --2- 172.21.15.120:0/3910515716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d64072540 0x7f4d640729b0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c009d90 tx=0x7f4d5c02beb0 comp rx=0 tx=0).stop 2024-02-06T22:18:07.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.995+0000 7f4d6907b640 1 -- 172.21.15.120:0/3910515716 shutdown_connections 2024-02-06T22:18:07.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.995+0000 7f4d6907b640 1 --2- 172.21.15.120:0/3910515716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d64072540 0x7f4d640729b0 unknown :-1 s=CLOSED pgs=3362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:07.000 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.995+0000 7f4d6907b640 1 -- 172.21.15.120:0/3910515716 >> 172.21.15.120:0/3910515716 conn(0x7f4d640b0bc0 msgr2=0x7f4d640b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:07.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.995+0000 7f4d6907b640 1 -- 172.21.15.120:0/3910515716 shutdown_connections 2024-02-06T22:18:07.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.995+0000 7f4d6907b640 1 -- 172.21.15.120:0/3910515716 wait complete. 2024-02-06T22:18:07.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.996+0000 7f4d6907b640 10 monclient: build_initial_monmap 2024-02-06T22:18:07.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.996+0000 7f4d6907b640 10 monclient: monmap: 2024-02-06T22:18:07.001 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:07.001 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:07.001 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:07.001 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:07.001 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:07.001 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:07.001 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:07.001 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:07.001 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:07.001 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:07.001 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.996+0000 7f4d6907b640 1 librados: starting msgr at 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.996+0000 7f4d6907b640 1 librados: starting objecter 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.996+0000 7f4d6907b640 1 Processor -- start 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.996+0000 7f4d6907b640 1 -- start start 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.996+0000 7f4d6907b640 1 librados: setting wanted keys 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.996+0000 7f4d6907b640 1 librados: calling monclient init 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.996+0000 7f4d6907b640 10 monclient: init 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 10 monclient: _add_conns ranks=[1,0,2] 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d64072540 0x7f4d640af400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 10 monclient(hunting): picked mon.noname-b con 0x7f4d64072540 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d640ae8e0 0x7f4d641d7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 10 monclient(hunting): picked mon.noname-a con 0x7f4d640ae8e0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d641d7dd0 0x7f4d64137d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 10 monclient(hunting): picked mon.noname-c con 0x7f4d641d7dd0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:07.002 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d641d83c0 con 0x7f4d640ae8e0 2024-02-06T22:18:07.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:07.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6413c8c0 con 0x7f4d64072540 2024-02-06T22:18:07.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:07.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d641382b0 con 0x7f4d641d7dd0 2024-02-06T22:18:07.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:07.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6907b640 10 monclient(hunting): authenticate will time out at 3517.333889s 2024-02-06T22:18:07.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d5bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d640ae8e0 0x7f4d641d7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:07.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.997+0000 7f4d6359e640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d641d7dd0 0x7f4d64137d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:07.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d5bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d640ae8e0 0x7f4d641d7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:49700/0 (socket says 172.21.15.120:49700) 2024-02-06T22:18:07.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d5bfff640 1 -- 172.21.15.120:0/2449892832 learned_addr learned my addr 172.21.15.120:0/2449892832 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:07.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d5bfff640 10 monclient(hunting): get_auth_request con 0x7f4d640ae8e0 auth_method 0 2024-02-06T22:18:07.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:07.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d5bfff640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:07.003 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:07.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d6359e640 10 monclient(hunting): get_auth_request con 0x7f4d641d7dd0 auth_method 0 2024-02-06T22:18:07.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d6359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:07.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d6359e640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:07.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d6359e640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:07.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:07.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:07.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:07.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d6359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:07.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d6359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:07.004 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:06.998+0000 7f4d6359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:07.005 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.001+0000 7f4d5bfff640 10 monclient(hunting): handle_auth_done global_id 16590 payload 824 2024-02-06T22:18:07.005 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.001+0000 7f4d5bfff640 1 -- 172.21.15.120:0/2449892832 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d641d7dd0 msgr2=0x7f4d64137d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:07.005 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.001+0000 7f4d5bfff640 1 --2- 172.21.15.120:0/2449892832 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d641d7dd0 0x7f4d64137d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:07.005 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.001+0000 7f4d5bfff640 1 -- 172.21.15.120:0/2449892832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d64072540 msgr2=0x7f4d640af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:07.005 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.001+0000 7f4d5bfff640 1 --2- 172.21.15.120:0/2449892832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d64072540 0x7f4d640af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:07.005 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.001+0000 7f4d5bfff640 10 monclient: _finish_hunting 0 2024-02-06T22:18:07.005 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.001+0000 7f4d5bfff640 1 monclient: found mon.noname-a 2024-02-06T22:18:07.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.001+0000 7f4d5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:07.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.001+0000 7f4d5bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:18:07.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.001+0000 7f4d5bfff640 1 -- 172.21.15.120:0/2449892832 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5c009a70 con 0x7f4d640ae8e0 2024-02-06T22:18:07.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.003+0000 7f4d5bfff640 1 --2- 172.21.15.120:0/2449892832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d640ae8e0 0x7f4d641d7890 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00c440 tx=0x7f4d4c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:07.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.003+0000 7f4d5b7fe640 10 client.?.objecter ms_handle_connect 0x7f4d640ae8e0 2024-02-06T22:18:07.006 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.003+0000 7f4d5b7fe640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:07.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d5b7fe640 1 -- 172.21.15.120:0/2449892832 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c0052d0 con 0x7f4d640ae8e0 2024-02-06T22:18:07.007 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:07.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d5b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:18:07.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d5b7fe640 10 monclient: dump: 2024-02-06T22:18:07.008 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:07.008 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:07.008 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:07.008 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:07.008 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:07.008 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:07.008 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:07.008 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:07.008 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:07.008 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:07.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d5b7fe640 10 monclient: _finish_auth 0 2024-02-06T22:18:07.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d5b7fe640 10 monclient: _check_auth_tickets 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d5b7fe640 1 -- 172.21.15.120:0/2449892832 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4c005430 con 0x7f4d640ae8e0 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d5b7fe640 1 -- 172.21.15.120:0/2449892832 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c007260 con 0x7f4d640ae8e0 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d5b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d5b7fe640 10 monclient: dump: 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:07.009 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:07.010 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:07.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d6907b640 5 monclient: authenticate success, global_id 16590 2024-02-06T22:18:07.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d61d9b640 4 set_mon_vals no callback set 2024-02-06T22:18:07.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d6907b640 10 monclient: _renew_subs 2024-02-06T22:18:07.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d6907b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:07.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d6907b640 1 -- 172.21.15.120:0/2449892832 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d64138480 con 0x7f4d640ae8e0 2024-02-06T22:18:07.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d6907b640 10 client.16590.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:07.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d6907b640 10 monclient: _renew_subs 2024-02-06T22:18:07.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d6907b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:07.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d6907b640 1 -- 172.21.15.120:0/2449892832 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d6413adf0 con 0x7f4d640ae8e0 2024-02-06T22:18:07.010 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.006+0000 7f4d6907b640 1 librados: init done 2024-02-06T22:18:07.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.008+0000 7f4d6907b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:07.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.008+0000 7f4d6907b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:07.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.008+0000 7f4d6907b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:07.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.008+0000 7f4d6907b640 1 -- 172.21.15.120:0/2449892832 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d641d81b0 con 0x7f4d640ae8e0 2024-02-06T22:18:07.025 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.025+0000 7f4d5b7fe640 1 -- 172.21.15.120:0/2449892832 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4d4c0073c0 con 0x7f4d640ae8e0 2024-02-06T22:18:07.026 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.025+0000 7f4d5b7fe640 1 --2- 172.21.15.120:0/2449892832 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f4d4406d1f0 0x7f4d4406f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:07.026 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.026+0000 7f4d62d9d640 1 --2- 172.21.15.120:0/2449892832 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f4d4406d1f0 0x7f4d4406f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:07.026 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.026+0000 7f4d62d9d640 10 monclient: get_auth_request con 0x7f4d4406d1f0 auth_method 0 2024-02-06T22:18:07.026 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.026+0000 7f4d62d9d640 1 --2- 172.21.15.120:0/2449892832 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f4d4406d1f0 0x7f4d4406f6b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4d54000f60 tx=0x7f4d540078f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:07.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.026+0000 7f4d5b7fe640 10 client.16590.objecter ms_handle_connect 0x7f4d4406d1f0 2024-02-06T22:18:07.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.028+0000 7f4d5b7fe640 1 -- 172.21.15.120:0/2449892832 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f4d4c058580 con 0x7f4d640ae8e0 2024-02-06T22:18:07.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.028+0000 7f4d5b7fe640 10 client.16590.objecter ms_dispatch 0x7f4d64000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:07.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.028+0000 7f4d5b7fe640 3 client.16590.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:07.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.028+0000 7f4d5b7fe640 3 client.16590.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:07.029 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.028+0000 7f4d5b7fe640 20 client.16590.objecter dump_active .. 0 homeless 2024-02-06T22:18:07.278 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.278+0000 7f4d5b7fe640 1 -- 172.21.15.120:0/2449892832 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4d4c054820 con 0x7f4d640ae8e0 2024-02-06T22:18:07.279 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.278+0000 7f4d5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:07.279 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.278+0000 7f4d5b7fe640 10 monclient: _finish_command 1 = system:0 2024-02-06T22:18:07.461 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.461+0000 7f4d6907b640 1 -- 172.21.15.120:0/2449892832 --> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d641d8500 con 0x7f4d4406d1f0 2024-02-06T22:18:07.462 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.461+0000 7f4d5b7fe640 1 -- 172.21.15.120:0/2449892832 <== mgr.4100 v2:172.21.15.135:6800/315664144 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4d641d8500 con 0x7f4d4406d1f0 2024-02-06T22:18:07.462 INFO:teuthology.orchestra.run.smithi120.stderr:instructed osd(s) 7 to deep-scrub 2024-02-06T22:18:07.464 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 10 librados: watch_flush enter 2024-02-06T22:18:07.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 10 librados: watch_flush exit 2024-02-06T22:18:07.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 20 client.16590.objecter shutdown clearing up homeless session... 2024-02-06T22:18:07.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 10 client.16590.objecter successfully canceled tick 2024-02-06T22:18:07.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 1 -- 172.21.15.120:0/2449892832 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f4d4406d1f0 msgr2=0x7f4d4406f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:07.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 1 --2- 172.21.15.120:0/2449892832 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f4d4406d1f0 0x7f4d4406f6b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4d54000f60 tx=0x7f4d540078f0 comp rx=0 tx=0).stop 2024-02-06T22:18:07.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 10 monclient: shutdown 2024-02-06T22:18:07.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:07.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 1 -- 172.21.15.120:0/2449892832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d640ae8e0 msgr2=0x7f4d641d7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:07.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 1 --2- 172.21.15.120:0/2449892832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d640ae8e0 0x7f4d641d7890 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00c440 tx=0x7f4d4c00c900 comp rx=0 tx=0).stop 2024-02-06T22:18:07.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 1 -- 172.21.15.120:0/2449892832 shutdown_connections 2024-02-06T22:18:07.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 1 --2- 172.21.15.120:0/2449892832 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f4d4406d1f0 0x7f4d4406f6b0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:07.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 1 --2- 172.21.15.120:0/2449892832 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d641d7dd0 0x7f4d64137d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:07.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 1 --2- 172.21.15.120:0/2449892832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d640ae8e0 0x7f4d641d7890 unknown :-1 s=CLOSED pgs=3363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:07.465 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 1 --2- 172.21.15.120:0/2449892832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d64072540 0x7f4d640af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:07.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 1 -- 172.21.15.120:0/2449892832 >> 172.21.15.120:0/2449892832 conn(0x7f4d640b0bc0 msgr2=0x7f4d6406f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:07.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 1 -- 172.21.15.120:0/2449892832 shutdown_connections 2024-02-06T22:18:07.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 1 -- 172.21.15.120:0/2449892832 wait complete. 2024-02-06T22:18:07.466 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.463+0000 7f4d6907b640 1 librados: shutdown 2024-02-06T22:18:07.475 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-06T22:18:07.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.585+0000 7f9dca0ff640 10 monclient: get_monmap_and_config 2024-02-06T22:18:07.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.585+0000 7f9dca0ff640 10 monclient: build_initial_monmap 2024-02-06T22:18:07.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.585+0000 7f9dca0ff640 10 monclient: monmap: 2024-02-06T22:18:07.586 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:07.586 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:07.586 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:07.586099+0000 2024-02-06T22:18:07.586 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:07.586099+0000 2024-02-06T22:18:07.587 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:07.587 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:07.587 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:07.587 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:07.587 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:07.587 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:07.587 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:07.587 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:07.589 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.586+0000 7f9dca0ff640 1 Processor -- start 2024-02-06T22:18:07.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.586+0000 7f9dca0ff640 1 -- start start 2024-02-06T22:18:07.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.586+0000 7f9dca0ff640 10 monclient: init 2024-02-06T22:18:07.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.587+0000 7f9dca0ff640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:07.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.587+0000 7f9dca0ff640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-02-06T22:18:07.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.587+0000 7f9dca0ff640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9dc40b54b0 0x7f9dc40a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:07.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.587+0000 7f9dca0ff640 10 monclient(hunting): picked mon.noname-a con 0x7f9dc40b54b0 addr v2:172.21.15.120:3300/0 2024-02-06T22:18:07.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.587+0000 7f9dca0ff640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9dc413f830 0x7f9dc413fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:07.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.587+0000 7f9dca0ff640 10 monclient(hunting): picked mon.noname-c con 0x7f9dc413f830 addr v2:172.21.15.135:3300/0 2024-02-06T22:18:07.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.587+0000 7f9dca0ff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9dc4142680 addr v1:172.21.15.120:6789/0 2024-02-06T22:18:07.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.587+0000 7f9dca0ff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc4143200 con 0x7f9dc4142680 2024-02-06T22:18:07.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.587+0000 7f9dca0ff640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:07.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.587+0000 7f9dca0ff640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc40ab520 con 0x7f9dc40b54b0 2024-02-06T22:18:07.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.587+0000 7f9dca0ff640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:07.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.587+0000 7f9dca0ff640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc40b5890 con 0x7f9dc413f830 2024-02-06T22:18:07.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.587+0000 7f9dca0ff640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:07.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.587+0000 7f9dca0ff640 10 monclient(hunting): authenticate will time out at 3517.923753s 2024-02-06T22:18:07.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.588+0000 7f9dc2ffd640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9dc413f830 0x7f9dc413fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:07.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.588+0000 7f9dc37fe640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9dc40b54b0 0x7f9dc40a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:07.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.588+0000 7f9dc3fff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f9dc4142680 0x7f9dc4142af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47540/0 (socket says 172.21.15.120:47540) 2024-02-06T22:18:07.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.588+0000 7f9dc3fff640 1 -- 172.21.15.120:0/4088922776 learned_addr learned my addr 172.21.15.120:0/4088922776 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T22:18:07.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.588+0000 7f9dc2ffd640 10 monclient(hunting): get_auth_request con 0x7f9dc413f830 auth_method 0 2024-02-06T22:18:07.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.588+0000 7f9dc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:07.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.588+0000 7f9dc2ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:07.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.588+0000 7f9dc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:07.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.588+0000 7f9dc37fe640 10 monclient(hunting): get_auth_request con 0x7f9dc40b54b0 auth_method 0 2024-02-06T22:18:07.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.588+0000 7f9dc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:07.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.588+0000 7f9dc37fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:07.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.588+0000 7f9dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:07.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.588+0000 7f9dc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:07.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.588+0000 7f9dc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:07.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.588+0000 7f9dc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:07.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.589+0000 7f9dc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:07.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.589+0000 7f9dc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:07.592 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.589+0000 7f9dc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:07.593 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.590+0000 7f9dc2ffd640 10 monclient(hunting): handle_auth_done global_id 21103 payload 274 2024-02-06T22:18:07.593 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.590+0000 7f9dc2ffd640 1 -- 172.21.15.120:0/4088922776 >> v2:172.21.15.120:3300/0 conn(0x7f9dc40b54b0 msgr2=0x7f9dc40a9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:07.593 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.590+0000 7f9dc2ffd640 1 --2- 172.21.15.120:0/4088922776 >> v2:172.21.15.120:3300/0 conn(0x7f9dc40b54b0 0x7f9dc40a9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:07.593 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.590+0000 7f9dc2ffd640 1 -- 172.21.15.120:0/4088922776 >> v1:172.21.15.120:6789/0 conn(0x7f9dc4142680 legacy=0x7f9dc4142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:07.593 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.590+0000 7f9dc2ffd640 10 monclient: _finish_hunting 0 2024-02-06T22:18:07.593 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.590+0000 7f9dc2ffd640 1 monclient: found mon.noname-c 2024-02-06T22:18:07.593 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.590+0000 7f9dc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:07.593 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.590+0000 7f9dc2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2024-02-06T22:18:07.594 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.590+0000 7f9dc2ffd640 1 -- 172.21.15.120:0/4088922776 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc40ab720 con 0x7f9dc413f830 2024-02-06T22:18:07.594 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.592+0000 7f9dc2ffd640 1 --2- 172.21.15.120:0/4088922776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dc413f830 0x7f9dc413fc70 secure :-1 s=READY pgs=5103 cs=0 l=1 rev1=1 crypto rx=0x7f9dc406deb0 tx=0x7f9db802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46208fe0ef0b0892 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:07.596 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dc27fc640 1 -- 172.21.15.120:0/4088922776 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db8031830 con 0x7f9dc413f830 2024-02-06T22:18:07.596 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:07.596 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dc27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-02-06T22:18:07.596 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dc27fc640 10 monclient: dump: 2024-02-06T22:18:07.596 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:07.596 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:07.596 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:07.596 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:07.596 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:07.597 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:07.597 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:07.597 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:07.597 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:07.597 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:07.597 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dc27fc640 10 monclient: _finish_auth 0 2024-02-06T22:18:07.597 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dc27fc640 10 monclient: _check_auth_tickets 2024-02-06T22:18:07.597 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:07.597 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dc27fc640 1 -- 172.21.15.120:0/4088922776 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db8031990 con 0x7f9dc413f830 2024-02-06T22:18:07.597 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dc27fc640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:07.597 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dc27fc640 1 -- 172.21.15.120:0/4088922776 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db8031d20 con 0x7f9dc413f830 2024-02-06T22:18:07.597 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dca0ff640 5 monclient: authenticate success, global_id 21103 2024-02-06T22:18:07.597 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dca0ff640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:07.597 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dca0ff640 4 set_mon_vals no callback set 2024-02-06T22:18:07.598 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:07.598 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dc27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T22:18:07.598 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.595+0000 7f9dc27fc640 10 monclient: dump: 2024-02-06T22:18:07.598 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:07.598 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:07.598 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:07.598 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:07.598 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:07.598 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:07.598 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:07.598 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:07.598 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:07.598 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:07.602 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.596+0000 7f9dca0ff640 10 monclient: shutdown 2024-02-06T22:18:07.602 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.596+0000 7f9dca0ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:07.602 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.596+0000 7f9dca0ff640 1 -- 172.21.15.120:0/4088922776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dc413f830 msgr2=0x7f9dc413fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:07.602 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.596+0000 7f9dca0ff640 1 --2- 172.21.15.120:0/4088922776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dc413f830 0x7f9dc413fc70 secure :-1 s=READY pgs=5103 cs=0 l=1 rev1=1 crypto rx=0x7f9dc406deb0 tx=0x7f9db802b8d0 comp rx=0 tx=0).stop 2024-02-06T22:18:07.602 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.596+0000 7f9dca0ff640 1 -- 172.21.15.120:0/4088922776 shutdown_connections 2024-02-06T22:18:07.602 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.596+0000 7f9dca0ff640 1 --2- 172.21.15.120:0/4088922776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dc413f830 0x7f9dc413fc70 unknown :-1 s=CLOSED pgs=5103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:07.602 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.596+0000 7f9dca0ff640 1 --2- 172.21.15.120:0/4088922776 >> v2:172.21.15.120:3300/0 conn(0x7f9dc40b54b0 0x7f9dc40a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:07.602 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.596+0000 7f9dca0ff640 1 -- 172.21.15.120:0/4088922776 >> 172.21.15.120:0/4088922776 conn(0x7f9dc40b8660 msgr2=0x7f9dc40b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:07.602 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.596+0000 7f9dca0ff640 1 -- 172.21.15.120:0/4088922776 shutdown_connections 2024-02-06T22:18:07.602 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.596+0000 7f9dca0ff640 1 -- 172.21.15.120:0/4088922776 wait complete. 2024-02-06T22:18:07.602 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.597+0000 7f9dca0ff640 10 monclient: build_initial_monmap 2024-02-06T22:18:07.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.597+0000 7f9dca0ff640 10 monclient: monmap: 2024-02-06T22:18:07.603 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:07.603 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:07.603 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:07.603 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:07.603 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:07.603 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:07.603 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:07.603 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:07.603 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:07.603 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:07.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.597+0000 7f9dca0ff640 1 librados: starting msgr at 2024-02-06T22:18:07.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.597+0000 7f9dca0ff640 1 librados: starting objecter 2024-02-06T22:18:07.603 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.597+0000 7f9dca0ff640 1 Processor -- start 2024-02-06T22:18:07.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.597+0000 7f9dca0ff640 1 -- start start 2024-02-06T22:18:07.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.597+0000 7f9dca0ff640 1 librados: setting wanted keys 2024-02-06T22:18:07.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.597+0000 7f9dca0ff640 1 librados: calling monclient init 2024-02-06T22:18:07.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.597+0000 7f9dca0ff640 10 monclient: init 2024-02-06T22:18:07.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:07.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-02-06T22:18:07.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dc40b54b0 0x7f9dc40b7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:07.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 10 monclient(hunting): picked mon.noname-a con 0x7f9dc40b54b0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:07.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc413f830 0x7f9dc4134fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:07.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 10 monclient(hunting): picked mon.noname-c con 0x7f9dc413f830 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:07.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dc4142680 0x7f9dc4133070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:07.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 10 monclient(hunting): picked mon.noname-b con 0x7f9dc4142680 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:07.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:07.604 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc41364b0 con 0x7f9dc40b54b0 2024-02-06T22:18:07.605 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:07.605 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc4136720 con 0x7f9dc4142680 2024-02-06T22:18:07.605 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:07.605 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc4137cd0 con 0x7f9dc413f830 2024-02-06T22:18:07.605 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:07.605 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dca0ff640 10 monclient(hunting): authenticate will time out at 3517.935018s 2024-02-06T22:18:07.605 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.598+0000 7f9dc37fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dc40b54b0 0x7f9dc40b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:07.605 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc3fff640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dc4142680 0x7f9dc4133070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:07.605 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc2ffd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc413f830 0x7f9dc4134fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:07.605 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc37fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dc40b54b0 0x7f9dc40b7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:49714/0 (socket says 172.21.15.120:49714) 2024-02-06T22:18:07.605 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc37fe640 1 -- 172.21.15.120:0/597926314 learned_addr learned my addr 172.21.15.120:0/597926314 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:07.605 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc37fe640 10 monclient(hunting): get_auth_request con 0x7f9dc40b54b0 auth_method 0 2024-02-06T22:18:07.605 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:07.605 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc37fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc3fff640 10 monclient(hunting): get_auth_request con 0x7f9dc4142680 auth_method 0 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc3fff640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc2ffd640 10 monclient(hunting): get_auth_request con 0x7f9dc413f830 auth_method 0 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc2ffd640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.599+0000 7f9dc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.600+0000 7f9dc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:07.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.600+0000 7f9dc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:07.607 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.600+0000 7f9dc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:07.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.602+0000 7f9dc3fff640 10 monclient(hunting): handle_auth_done global_id 21106 payload 824 2024-02-06T22:18:07.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.602+0000 7f9dc3fff640 1 -- 172.21.15.120:0/597926314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc413f830 msgr2=0x7f9dc4134fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:07.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.602+0000 7f9dc3fff640 1 --2- 172.21.15.120:0/597926314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc413f830 0x7f9dc4134fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:07.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.602+0000 7f9dc3fff640 1 -- 172.21.15.120:0/597926314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dc40b54b0 msgr2=0x7f9dc40b7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:07.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.602+0000 7f9dc3fff640 1 --2- 172.21.15.120:0/597926314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dc40b54b0 0x7f9dc40b7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:07.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.602+0000 7f9dc3fff640 10 monclient: _finish_hunting 0 2024-02-06T22:18:07.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.602+0000 7f9dc3fff640 1 monclient: found mon.noname-b 2024-02-06T22:18:07.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.602+0000 7f9dc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:07.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.602+0000 7f9dc3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2024-02-06T22:18:07.609 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.602+0000 7f9dc3fff640 1 -- 172.21.15.120:0/597926314 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db8009400 con 0x7f9dc4142680 2024-02-06T22:18:07.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.604+0000 7f9dc3fff640 1 --2- 172.21.15.120:0/597926314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dc4142680 0x7f9dc4133070 secure :-1 s=READY pgs=5104 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc00caa0 tx=0x7f9dbc00cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:07.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.604+0000 7f9dabfff640 10 client.?.objecter ms_handle_connect 0x7f9dc4142680 2024-02-06T22:18:07.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.604+0000 7f9dabfff640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:07.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dabfff640 1 -- 172.21.15.120:0/597926314 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc00ab40 con 0x7f9dc4142680 2024-02-06T22:18:07.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:07.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dabfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:18:07.610 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dabfff640 10 monclient: dump: 2024-02-06T22:18:07.610 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:07.610 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:07.610 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:07.610 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:07.610 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:07.611 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:07.611 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:07.611 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:07.611 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:07.611 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:07.611 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dabfff640 10 monclient: _finish_auth 0 2024-02-06T22:18:07.611 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dabfff640 10 monclient: _check_auth_tickets 2024-02-06T22:18:07.611 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:07.611 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dabfff640 1 -- 172.21.15.120:0/597926314 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dbc00aca0 con 0x7f9dc4142680 2024-02-06T22:18:07.611 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dabfff640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:07.611 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dabfff640 1 -- 172.21.15.120:0/597926314 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc012600 con 0x7f9dc4142680 2024-02-06T22:18:07.612 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dca0ff640 5 monclient: authenticate success, global_id 21106 2024-02-06T22:18:07.612 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:07.612 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dabfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-02-06T22:18:07.612 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dabfff640 10 monclient: dump: 2024-02-06T22:18:07.612 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:07.612 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:07.612 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:07.612 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:07.612 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:07.612 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:07.612 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:07.612 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:07.612 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:07.612 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:07.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dca0ff640 10 monclient: _renew_subs 2024-02-06T22:18:07.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dc1ffb640 4 set_mon_vals no callback set 2024-02-06T22:18:07.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dca0ff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:07.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.607+0000 7f9dca0ff640 1 -- 172.21.15.120:0/597926314 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc4136bb0 con 0x7f9dc4142680 2024-02-06T22:18:07.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.608+0000 7f9dca0ff640 10 client.21106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:07.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.608+0000 7f9dca0ff640 10 monclient: _renew_subs 2024-02-06T22:18:07.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.608+0000 7f9dca0ff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:07.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.608+0000 7f9dca0ff640 1 -- 172.21.15.120:0/597926314 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc4138640 con 0x7f9dc4142680 2024-02-06T22:18:07.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.608+0000 7f9dca0ff640 1 librados: init done 2024-02-06T22:18:07.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.608+0000 7f9dca0ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:07.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.608+0000 7f9dca0ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:07.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.608+0000 7f9dca0ff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2024-02-06T22:18:07.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.608+0000 7f9dca0ff640 1 -- 172.21.15.120:0/597926314 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc404ef90 con 0x7f9dc4142680 2024-02-06T22:18:07.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.626+0000 7f9dabfff640 1 -- 172.21.15.120:0/597926314 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9dbc007070 con 0x7f9dc4142680 2024-02-06T22:18:07.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.627+0000 7f9dabfff640 1 --2- 172.21.15.120:0/597926314 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f9db006d1d0 0x7f9db006f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:07.628 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.627+0000 7f9dc37fe640 1 --2- 172.21.15.120:0/597926314 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f9db006d1d0 0x7f9db006f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:07.628 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.627+0000 7f9dc37fe640 10 monclient: get_auth_request con 0x7f9db006d1d0 auth_method 0 2024-02-06T22:18:07.628 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.627+0000 7f9dc37fe640 1 --2- 172.21.15.120:0/597926314 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f9db006d1d0 0x7f9db006f690 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f9db4009c00 tx=0x7f9db4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:07.628 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.627+0000 7f9dabfff640 10 client.21106.objecter ms_handle_connect 0x7f9db006d1d0 2024-02-06T22:18:07.629 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.628+0000 7f9dabfff640 1 -- 172.21.15.120:0/597926314 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f9dbc058cc0 con 0x7f9dc4142680 2024-02-06T22:18:07.629 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.628+0000 7f9dabfff640 10 client.21106.objecter ms_dispatch 0x7f9dc4000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:07.629 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.628+0000 7f9dabfff640 3 client.21106.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:07.629 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.628+0000 7f9dabfff640 3 client.21106.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:07.629 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.628+0000 7f9dabfff640 20 client.21106.objecter dump_active .. 0 homeless 2024-02-06T22:18:07.868 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.868+0000 7f9dabfff640 1 -- 172.21.15.120:0/597926314 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9dbc054f60 con 0x7f9dc4142680 2024-02-06T22:18:07.868 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.868+0000 7f9dabfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:07.868 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:07.868+0000 7f9dabfff640 10 monclient: _finish_command 1 = system:0 2024-02-06T22:18:08.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.038+0000 7f9dca0ff640 1 -- 172.21.15.120:0/597926314 --> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9dc4138880 con 0x7f9db006d1d0 2024-02-06T22:18:08.043 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.042+0000 7f9dabfff640 1 -- 172.21.15.120:0/597926314 <== mgr.4100 v2:172.21.15.135:6800/315664144 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+150907 (secure 0 0 0) 0x7f9db4011b20 con 0x7f9db006d1d0 2024-02-06T22:18:08.044 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T22:18:08.045 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2024-02-06T22:18:08.047 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 10 librados: watch_flush enter 2024-02-06T22:18:08.047 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 10 librados: watch_flush exit 2024-02-06T22:18:08.047 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 20 client.21106.objecter shutdown clearing up homeless session... 2024-02-06T22:18:08.047 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 10 client.21106.objecter successfully canceled tick 2024-02-06T22:18:08.047 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 1 -- 172.21.15.120:0/597926314 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f9db006d1d0 msgr2=0x7f9db006f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:08.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 1 --2- 172.21.15.120:0/597926314 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f9db006d1d0 0x7f9db006f690 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f9db4009c00 tx=0x7f9db4009190 comp rx=0 tx=0).stop 2024-02-06T22:18:08.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 10 monclient: shutdown 2024-02-06T22:18:08.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:08.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 1 -- 172.21.15.120:0/597926314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dc4142680 msgr2=0x7f9dc4133070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:08.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 1 --2- 172.21.15.120:0/597926314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dc4142680 0x7f9dc4133070 secure :-1 s=READY pgs=5104 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc00caa0 tx=0x7f9dbc00cf60 comp rx=0 tx=0).stop 2024-02-06T22:18:08.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 1 -- 172.21.15.120:0/597926314 shutdown_connections 2024-02-06T22:18:08.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 1 --2- 172.21.15.120:0/597926314 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f9db006d1d0 0x7f9db006f690 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:08.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 1 --2- 172.21.15.120:0/597926314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dc4142680 0x7f9dc4133070 unknown :-1 s=CLOSED pgs=5104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:08.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 1 --2- 172.21.15.120:0/597926314 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9dc413f830 0x7f9dc4134fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:08.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 1 --2- 172.21.15.120:0/597926314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dc40b54b0 0x7f9dc40b7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:08.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 1 -- 172.21.15.120:0/597926314 >> 172.21.15.120:0/597926314 conn(0x7f9dc40b8660 msgr2=0x7f9dc413a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:08.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 1 -- 172.21.15.120:0/597926314 shutdown_connections 2024-02-06T22:18:08.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 1 -- 172.21.15.120:0/597926314 wait complete. 2024-02-06T22:18:08.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:08.045+0000 7f9dca0ff640 1 librados: shutdown 2024-02-06T22:18:08.056 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":2453,"stamp":"2024-02-06T22:18:06.292080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":583324,"kb_used_data":40952,"kb_used_omap":12,"kb_used_meta":542259,"kb_avail":754391396,"statfs":{"total":773094113280,"available":772496789504,"internally_reserved":0,"allocated":41934848,"data_stored":19968162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":555273776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":122,"apply_latency_ms":122,"commit_latency_ns":122000000,"apply_latency_ns":122000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.895627"},"pg_stats":[{"pgid":"12.1c","version":"0'0","reported_seq":2594,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:59.140379+0000","last_change":"2024-02-06T21:42:50.083729+0000","last_active":"2024-02-06T22:17:59.140379+0000","last_peered":"2024-02-06T22:17:59.140379+0000","last_clean":"2024-02-06T22:17:59.140379+0000","last_became_active":"2024-02-06T21:42:50.078423+0000","last_became_peered":"2024-02-06T21:42:50.078423+0000","last_unstale":"2024-02-06T22:17:59.140379+0000","last_undegraded":"2024-02-06T22:17:59.140379+0000","last_fullsized":"2024-02-06T22:17:59.140379+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:44:16.757427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":2587,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:59.805735+0000","last_change":"2024-02-06T21:43:19.779411+0000","last_active":"2024-02-06T22:17:59.805735+0000","last_peered":"2024-02-06T22:17:59.805735+0000","last_clean":"2024-02-06T22:17:59.805735+0000","last_became_active":"2024-02-06T21:43:19.773686+0000","last_became_peered":"2024-02-06T21:43:19.773686+0000","last_unstale":"2024-02-06T22:17:59.805735+0000","last_undegraded":"2024-02-06T22:17:59.805735+0000","last_fullsized":"2024-02-06T22:17:59.805735+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:54:24.026659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":2602,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:03.440394+0000","last_change":"2024-02-06T22:18:03.441138+0000","last_active":"2024-02-06T22:18:03.440394+0000","last_peered":"2024-02-06T22:18:03.440394+0000","last_clean":"2024-02-06T22:18:03.440394+0000","last_became_active":"2024-02-06T21:42:53.291194+0000","last_became_peered":"2024-02-06T21:42:53.291194+0000","last_unstale":"2024-02-06T22:18:03.440394+0000","last_undegraded":"2024-02-06T22:18:03.440394+0000","last_fullsized":"2024-02-06T22:18:03.440394+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:03.438950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:03.438950+0000","last_clean_scrub_stamp":"2024-02-06T22:18:03.438950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T07:42:58.828481+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":2584,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:59.607252+0000","last_change":"2024-02-06T21:42:45.080259+0000","last_active":"2024-02-06T22:17:59.607252+0000","last_peered":"2024-02-06T22:17:59.607252+0000","last_clean":"2024-02-06T22:17:59.607252+0000","last_became_active":"2024-02-06T21:42:45.065933+0000","last_became_peered":"2024-02-06T21:42:45.065933+0000","last_unstale":"2024-02-06T22:17:59.607252+0000","last_undegraded":"2024-02-06T22:17:59.607252+0000","last_fullsized":"2024-02-06T22:17:59.607252+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":2580,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:00.700796+0000","last_change":"2024-02-06T21:42:58.554516+0000","last_active":"2024-02-06T22:18:00.700796+0000","last_peered":"2024-02-06T22:18:00.700796+0000","last_clean":"2024-02-06T22:18:00.700796+0000","last_became_active":"2024-02-06T21:42:58.548361+0000","last_became_peered":"2024-02-06T21:42:58.548361+0000","last_unstale":"2024-02-06T22:18:00.700796+0000","last_undegraded":"2024-02-06T22:18:00.700796+0000","last_fullsized":"2024-02-06T22:18:00.700796+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"124'7","reported_seq":2599,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:58.937278+0000","last_change":"2024-02-06T21:43:15.527724+0000","last_active":"2024-02-06T22:17:58.937278+0000","last_peered":"2024-02-06T22:17:58.937278+0000","last_clean":"2024-02-06T22:17:58.937278+0000","last_became_active":"2024-02-06T21:43:15.497081+0000","last_became_peered":"2024-02-06T21:43:15.497081+0000","last_unstale":"2024-02-06T22:17:58.937278+0000","last_undegraded":"2024-02-06T22:17:58.937278+0000","last_fullsized":"2024-02-06T22:17:58.937278+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":2595,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:58.985053+0000","last_change":"2024-02-06T21:43:13.230133+0000","last_active":"2024-02-06T22:17:58.985053+0000","last_peered":"2024-02-06T22:17:58.985053+0000","last_clean":"2024-02-06T22:17:58.985053+0000","last_became_active":"2024-02-06T21:43:13.108638+0000","last_became_peered":"2024-02-06T21:43:13.108638+0000","last_unstale":"2024-02-06T22:17:58.985053+0000","last_undegraded":"2024-02-06T22:17:58.985053+0000","last_fullsized":"2024-02-06T22:17:58.985053+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":2601,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:58.984324+0000","last_change":"2024-02-06T21:43:06.276185+0000","last_active":"2024-02-06T22:17:58.984324+0000","last_peered":"2024-02-06T22:17:58.984324+0000","last_clean":"2024-02-06T22:17:58.984324+0000","last_became_active":"2024-02-06T21:43:06.270947+0000","last_became_peered":"2024-02-06T21:43:06.270947+0000","last_unstale":"2024-02-06T22:17:58.984324+0000","last_undegraded":"2024-02-06T22:17:58.984324+0000","last_fullsized":"2024-02-06T22:17:58.984324+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:27:26.086535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":2595,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:58.998967+0000","last_change":"2024-02-06T21:42:53.563677+0000","last_active":"2024-02-06T22:17:58.998967+0000","last_peered":"2024-02-06T22:17:58.998967+0000","last_clean":"2024-02-06T22:17:58.998967+0000","last_became_active":"2024-02-06T21:42:53.535059+0000","last_became_peered":"2024-02-06T21:42:53.535059+0000","last_unstale":"2024-02-06T22:17:58.998967+0000","last_undegraded":"2024-02-06T22:17:58.998967+0000","last_fullsized":"2024-02-06T22:17:58.998967+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":2582,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:00.718818+0000","last_change":"2024-02-06T21:42:46.177846+0000","last_active":"2024-02-06T22:18:00.718818+0000","last_peered":"2024-02-06T22:18:00.718818+0000","last_clean":"2024-02-06T22:18:00.718818+0000","last_became_active":"2024-02-06T21:42:46.168640+0000","last_became_peered":"2024-02-06T21:42:46.168640+0000","last_unstale":"2024-02-06T22:18:00.718818+0000","last_undegraded":"2024-02-06T22:18:00.718818+0000","last_fullsized":"2024-02-06T22:18:00.718818+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":2578,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:59.799991+0000","last_change":"2024-02-06T21:43:27.891680+0000","last_active":"2024-02-06T22:17:59.799991+0000","last_peered":"2024-02-06T22:17:59.799991+0000","last_clean":"2024-02-06T22:17:59.799991+0000","last_became_active":"2024-02-06T21:43:27.883526+0000","last_became_peered":"2024-02-06T21:43:27.883526+0000","last_unstale":"2024-02-06T22:17:59.799991+0000","last_undegraded":"2024-02-06T22:17:59.799991+0000","last_fullsized":"2024-02-06T22:17:59.799991+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:21:41.402059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":2583,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:00.221941+0000","last_change":"2024-02-06T21:42:45.882433+0000","last_active":"2024-02-06T22:18:00.221941+0000","last_peered":"2024-02-06T22:18:00.221941+0000","last_clean":"2024-02-06T22:18:00.221941+0000","last_became_active":"2024-02-06T21:42:45.868987+0000","last_became_peered":"2024-02-06T21:42:45.868987+0000","last_unstale":"2024-02-06T22:18:00.221941+0000","last_undegraded":"2024-02-06T22:18:00.221941+0000","last_fullsized":"2024-02-06T22:18:00.221941+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T00:26:41.132150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":2581,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:59.618268+0000","last_change":"2024-02-06T21:42:59.955119+0000","last_active":"2024-02-06T22:17:59.618268+0000","last_peered":"2024-02-06T22:17:59.618268+0000","last_clean":"2024-02-06T22:17:59.618268+0000","last_became_active":"2024-02-06T21:42:59.949344+0000","last_became_peered":"2024-02-06T21:42:59.949344+0000","last_unstale":"2024-02-06T22:17:59.618268+0000","last_undegraded":"2024-02-06T22:17:59.618268+0000","last_fullsized":"2024-02-06T22:17:59.618268+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"1346'78","reported_seq":2715,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:03.290419+0000","last_change":"2024-02-06T22:18:03.296110+0000","last_active":"2024-02-06T22:18:03.290419+0000","last_peered":"2024-02-06T22:18:03.290419+0000","last_clean":"2024-02-06T22:18:03.290419+0000","last_became_active":"2024-02-06T21:41:31.305766+0000","last_became_peered":"2024-02-06T21:41:31.305766+0000","last_unstale":"2024-02-06T22:18:03.290419+0000","last_undegraded":"2024-02-06T22:18:03.290419+0000","last_fullsized":"2024-02-06T22:18:03.290419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"1346'78","last_scrub_stamp":"2024-02-06T22:18:03.279798+0000","last_deep_scrub":"1346'78","last_deep_scrub_stamp":"2024-02-06T22:18:03.279798+0000","last_clean_scrub_stamp":"2024-02-06T22:18:03.279798+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:53:47.078250+0000","scrub_duration":415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2595,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:59.602578+0000","last_change":"2024-02-06T21:41:36.168152+0000","last_active":"2024-02-06T22:17:59.602578+0000","last_peered":"2024-02-06T22:17:59.602578+0000","last_clean":"2024-02-06T22:17:59.602578+0000","last_became_active":"2024-02-06T21:41:36.162988+0000","last_became_peered":"2024-02-06T21:41:36.162988+0000","last_unstale":"2024-02-06T22:17:59.602578+0000","last_undegraded":"2024-02-06T22:17:59.602578+0000","last_fullsized":"2024-02-06T22:17:59.602578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":2582,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:00.246758+0000","last_change":"2024-02-06T21:42:56.031740+0000","last_active":"2024-02-06T22:18:00.246758+0000","last_peered":"2024-02-06T22:18:00.246758+0000","last_clean":"2024-02-06T22:18:00.246758+0000","last_became_active":"2024-02-06T21:42:56.023045+0000","last_became_peered":"2024-02-06T21:42:56.023045+0000","last_unstale":"2024-02-06T22:18:00.246758+0000","last_undegraded":"2024-02-06T22:18:00.246758+0000","last_fullsized":"2024-02-06T22:18:00.246758+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:19:09.477603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2597,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:59.591491+0000","last_change":"2024-02-06T21:41:35.688242+0000","last_active":"2024-02-06T22:17:59.591491+0000","last_peered":"2024-02-06T22:17:59.591491+0000","last_clean":"2024-02-06T22:17:59.591491+0000","last_became_active":"2024-02-06T21:41:35.642164+0000","last_became_peered":"2024-02-06T21:41:35.642164+0000","last_unstale":"2024-02-06T22:17:59.591491+0000","last_undegraded":"2024-02-06T22:17:59.591491+0000","last_fullsized":"2024-02-06T22:17:59.591491+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":2578,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:59.608879+0000","last_change":"2024-02-06T21:43:05.426147+0000","last_active":"2024-02-06T22:17:59.608879+0000","last_peered":"2024-02-06T22:17:59.608879+0000","last_clean":"2024-02-06T22:17:59.608879+0000","last_became_active":"2024-02-06T21:43:05.414230+0000","last_became_peered":"2024-02-06T21:43:05.414230+0000","last_unstale":"2024-02-06T22:17:59.608879+0000","last_undegraded":"2024-02-06T22:17:59.608879+0000","last_fullsized":"2024-02-06T22:17:59.608879+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2594,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:00.234941+0000","last_change":"2024-02-06T21:41:35.636555+0000","last_active":"2024-02-06T22:18:00.234941+0000","last_peered":"2024-02-06T22:18:00.234941+0000","last_clean":"2024-02-06T22:18:00.234941+0000","last_became_active":"2024-02-06T21:41:35.603135+0000","last_became_peered":"2024-02-06T21:41:35.603135+0000","last_unstale":"2024-02-06T22:18:00.234941+0000","last_undegraded":"2024-02-06T22:18:00.234941+0000","last_fullsized":"2024-02-06T22:18:00.234941+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T03:13:14.699865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":2583,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:00.242279+0000","last_change":"2024-02-06T21:42:49.418971+0000","last_active":"2024-02-06T22:18:00.242279+0000","last_peered":"2024-02-06T22:18:00.242279+0000","last_clean":"2024-02-06T22:18:00.242279+0000","last_became_active":"2024-02-06T21:42:49.409862+0000","last_became_peered":"2024-02-06T21:42:49.409862+0000","last_unstale":"2024-02-06T22:18:00.242279+0000","last_undegraded":"2024-02-06T22:18:00.242279+0000","last_fullsized":"2024-02-06T22:18:00.242279+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:08:26.530828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2596,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:59.598720+0000","last_change":"2024-02-06T21:41:35.892547+0000","last_active":"2024-02-06T22:17:59.598720+0000","last_peered":"2024-02-06T22:17:59.598720+0000","last_clean":"2024-02-06T22:17:59.598720+0000","last_became_active":"2024-02-06T21:41:35.877657+0000","last_became_peered":"2024-02-06T21:41:35.877657+0000","last_unstale":"2024-02-06T22:17:59.598720+0000","last_undegraded":"2024-02-06T22:17:59.598720+0000","last_fullsized":"2024-02-06T22:17:59.598720+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":2601,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:58.996851+0000","last_change":"2024-02-06T21:42:56.499357+0000","last_active":"2024-02-06T22:17:58.996851+0000","last_peered":"2024-02-06T22:17:58.996851+0000","last_clean":"2024-02-06T22:17:58.996851+0000","last_became_active":"2024-02-06T21:42:56.473273+0000","last_became_peered":"2024-02-06T21:42:56.473273+0000","last_unstale":"2024-02-06T22:17:58.996851+0000","last_undegraded":"2024-02-06T22:17:58.996851+0000","last_fullsized":"2024-02-06T22:17:58.996851+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:45:45.452902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2606,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:59.143007+0000","last_change":"2024-02-06T21:41:35.918723+0000","last_active":"2024-02-06T22:17:59.143007+0000","last_peered":"2024-02-06T22:17:59.143007+0000","last_clean":"2024-02-06T22:17:59.143007+0000","last_became_active":"2024-02-06T21:41:35.891748+0000","last_became_peered":"2024-02-06T21:41:35.891748+0000","last_unstale":"2024-02-06T22:17:59.143007+0000","last_undegraded":"2024-02-06T22:17:59.143007+0000","last_fullsized":"2024-02-06T22:17:59.143007+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T03:59:11.081636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":2583,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:00.236819+0000","last_change":"2024-02-06T21:42:49.677946+0000","last_active":"2024-02-06T22:18:00.236819+0000","last_peered":"2024-02-06T22:18:00.236819+0000","last_clean":"2024-02-06T22:18:00.236819+0000","last_became_active":"2024-02-06T21:42:49.671861+0000","last_became_peered":"2024-02-06T21:42:49.671861+0000","last_unstale":"2024-02-06T22:18:00.236819+0000","last_undegraded":"2024-02-06T22:18:00.236819+0000","last_fullsized":"2024-02-06T22:18:00.236819+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:43:48.362335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2595,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:59.619893+0000","last_change":"2024-02-06T21:41:35.766413+0000","last_active":"2024-02-06T22:17:59.619893+0000","last_peered":"2024-02-06T22:17:59.619893+0000","last_clean":"2024-02-06T22:17:59.619893+0000","last_became_active":"2024-02-06T21:41:35.761324+0000","last_became_peered":"2024-02-06T21:41:35.761324+0000","last_unstale":"2024-02-06T22:17:59.619893+0000","last_undegraded":"2024-02-06T22:17:59.619893+0000","last_fullsized":"2024-02-06T22:17:59.619893+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":2584,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:00.226015+0000","last_change":"2024-02-06T21:42:44.700963+0000","last_active":"2024-02-06T22:18:00.226015+0000","last_peered":"2024-02-06T22:18:00.226015+0000","last_clean":"2024-02-06T22:18:00.226015+0000","last_became_active":"2024-02-06T21:42:44.691217+0000","last_became_peered":"2024-02-06T21:42:44.691217+0000","last_unstale":"2024-02-06T22:18:00.226015+0000","last_undegraded":"2024-02-06T22:18:00.226015+0000","last_fullsized":"2024-02-06T22:18:00.226015+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:15:55.323353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2617,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:58.990711+0000","last_change":"2024-02-06T21:41:36.193098+0000","last_active":"2024-02-06T22:17:58.990711+0000","last_peered":"2024-02-06T22:17:58.990711+0000","last_clean":"2024-02-06T22:17:58.990711+0000","last_became_active":"2024-02-06T21:41:36.160648+0000","last_became_peered":"2024-02-06T21:41:36.160648+0000","last_unstale":"2024-02-06T22:17:58.990711+0000","last_undegraded":"2024-02-06T22:17:58.990711+0000","last_fullsized":"2024-02-06T22:17:58.990711+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T09:01:03.161738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":2588,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:59.791107+0000","last_change":"2024-02-06T21:42:52.618077+0000","last_active":"2024-02-06T22:17:59.791107+0000","last_peered":"2024-02-06T22:17:59.791107+0000","last_clean":"2024-02-06T22:17:59.791107+0000","last_became_active":"2024-02-06T21:42:52.611794+0000","last_became_peered":"2024-02-06T21:42:52.611794+0000","last_unstale":"2024-02-06T22:17:59.791107+0000","last_undegraded":"2024-02-06T22:17:59.791107+0000","last_fullsized":"2024-02-06T22:17:59.791107+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:25:35.938955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2592,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:59.610786+0000","last_change":"2024-02-06T21:41:36.144443+0000","last_active":"2024-02-06T22:17:59.610786+0000","last_peered":"2024-02-06T22:17:59.610786+0000","last_clean":"2024-02-06T22:17:59.610786+0000","last_became_active":"2024-02-06T21:41:36.139312+0000","last_became_peered":"2024-02-06T21:41:36.139312+0000","last_unstale":"2024-02-06T22:17:59.610786+0000","last_undegraded":"2024-02-06T22:17:59.610786+0000","last_fullsized":"2024-02-06T22:17:59.610786+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:41:34.053422+0000","last_clean_scrub_stamp":"2024-02-06T21:41:34.053422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":2581,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:00.722930+0000","last_change":"2024-02-06T21:42:49.714094+0000","last_active":"2024-02-06T22:18:00.722930+0000","last_peered":"2024-02-06T22:18:00.722930+0000","last_clean":"2024-02-06T22:18:00.722930+0000","last_became_active":"2024-02-06T21:42:49.708820+0000","last_became_peered":"2024-02-06T21:42:49.708820+0000","last_unstale":"2024-02-06T22:18:00.722930+0000","last_undegraded":"2024-02-06T22:18:00.722930+0000","last_fullsized":"2024-02-06T22:18:00.722930+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":2593,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:58.982055+0000","last_change":"2024-02-06T21:43:13.244098+0000","last_active":"2024-02-06T22:17:58.982055+0000","last_peered":"2024-02-06T22:17:58.982055+0000","last_clean":"2024-02-06T22:17:58.982055+0000","last_became_active":"2024-02-06T21:43:13.123786+0000","last_became_peered":"2024-02-06T21:43:13.123786+0000","last_unstale":"2024-02-06T22:17:58.982055+0000","last_undegraded":"2024-02-06T22:17:58.982055+0000","last_fullsized":"2024-02-06T22:17:58.982055+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":2578,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:00.233207+0000","last_change":"2024-02-06T21:42:56.872139+0000","last_active":"2024-02-06T22:18:00.233207+0000","last_peered":"2024-02-06T22:18:00.233207+0000","last_clean":"2024-02-06T22:18:00.233207+0000","last_became_active":"2024-02-06T21:42:56.865960+0000","last_became_peered":"2024-02-06T21:42:56.865960+0000","last_unstale":"2024-02-06T22:18:00.233207+0000","last_undegraded":"2024-02-06T22:18:00.233207+0000","last_fullsized":"2024-02-06T22:18:00.233207+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:13:16.665309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":2582,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:00.224083+0000","last_change":"2024-02-06T21:42:45.712485+0000","last_active":"2024-02-06T22:18:00.224083+0000","last_peered":"2024-02-06T22:18:00.224083+0000","last_clean":"2024-02-06T22:18:00.224083+0000","last_became_active":"2024-02-06T21:42:45.706432+0000","last_became_peered":"2024-02-06T21:42:45.706432+0000","last_unstale":"2024-02-06T22:18:00.224083+0000","last_undegraded":"2024-02-06T22:18:00.224083+0000","last_fullsized":"2024-02-06T22:18:00.224083+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:46:59.377729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":2580,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:58.944614+0000","last_change":"2024-02-06T21:43:08.879863+0000","last_active":"2024-02-06T22:17:58.944614+0000","last_peered":"2024-02-06T22:17:58.944614+0000","last_clean":"2024-02-06T22:17:58.944614+0000","last_became_active":"2024-02-06T21:43:08.871254+0000","last_became_peered":"2024-02-06T21:43:08.871254+0000","last_unstale":"2024-02-06T22:17:58.944614+0000","last_undegraded":"2024-02-06T22:17:58.944614+0000","last_fullsized":"2024-02-06T22:17:58.944614+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":2591,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:58.995605+0000","last_change":"2024-02-06T21:42:55.060952+0000","last_active":"2024-02-06T22:17:58.995605+0000","last_peered":"2024-02-06T22:17:58.995605+0000","last_clean":"2024-02-06T22:17:58.995605+0000","last_became_active":"2024-02-06T21:42:55.034062+0000","last_became_peered":"2024-02-06T21:42:55.034062+0000","last_unstale":"2024-02-06T22:17:58.995605+0000","last_undegraded":"2024-02-06T22:17:58.995605+0000","last_fullsized":"2024-02-06T22:17:58.995605+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":2583,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:58.941511+0000","last_change":"2024-02-06T21:43:15.867032+0000","last_active":"2024-02-06T22:17:58.941511+0000","last_peered":"2024-02-06T22:17:58.941511+0000","last_clean":"2024-02-06T22:17:58.941511+0000","last_became_active":"2024-02-06T21:43:15.861140+0000","last_became_peered":"2024-02-06T21:43:15.861140+0000","last_unstale":"2024-02-06T22:17:58.941511+0000","last_undegraded":"2024-02-06T22:17:58.941511+0000","last_fullsized":"2024-02-06T22:17:58.941511+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":2582,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:00.713873+0000","last_change":"2024-02-06T21:42:55.175403+0000","last_active":"2024-02-06T22:18:00.713873+0000","last_peered":"2024-02-06T22:18:00.713873+0000","last_clean":"2024-02-06T22:18:00.713873+0000","last_became_active":"2024-02-06T21:42:55.168677+0000","last_became_peered":"2024-02-06T21:42:55.168677+0000","last_unstale":"2024-02-06T22:18:00.713873+0000","last_undegraded":"2024-02-06T22:18:00.713873+0000","last_fullsized":"2024-02-06T22:18:00.713873+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":2596,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:59.135204+0000","last_change":"2024-02-06T21:42:52.000053+0000","last_active":"2024-02-06T22:17:59.135204+0000","last_peered":"2024-02-06T22:17:59.135204+0000","last_clean":"2024-02-06T22:17:59.135204+0000","last_became_active":"2024-02-06T21:42:51.993812+0000","last_became_peered":"2024-02-06T21:42:51.993812+0000","last_unstale":"2024-02-06T22:17:59.135204+0000","last_undegraded":"2024-02-06T22:17:59.135204+0000","last_fullsized":"2024-02-06T22:17:59.135204+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:44:21.306346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":2580,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:58.995121+0000","last_change":"2024-02-06T21:44:01.005089+0000","last_active":"2024-02-06T22:17:58.995121+0000","last_peered":"2024-02-06T22:17:58.995121+0000","last_clean":"2024-02-06T22:17:58.995121+0000","last_became_active":"2024-02-06T21:44:00.992836+0000","last_became_peered":"2024-02-06T21:44:00.992836+0000","last_unstale":"2024-02-06T22:17:58.995121+0000","last_undegraded":"2024-02-06T22:17:58.995121+0000","last_fullsized":"2024-02-06T22:17:58.995121+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:34:43.433551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":2589,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:58.975462+0000","last_change":"2024-02-06T21:43:14.066712+0000","last_active":"2024-02-06T22:17:58.975462+0000","last_peered":"2024-02-06T22:17:58.975462+0000","last_clean":"2024-02-06T22:17:58.975462+0000","last_became_active":"2024-02-06T21:43:14.057258+0000","last_became_peered":"2024-02-06T21:43:14.057258+0000","last_unstale":"2024-02-06T22:17:58.975462+0000","last_undegraded":"2024-02-06T22:17:58.975462+0000","last_fullsized":"2024-02-06T22:17:58.975462+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":2588,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:17:59.792949+0000","last_change":"2024-02-06T21:42:49.763089+0000","last_active":"2024-02-06T22:17:59.792949+0000","last_peered":"2024-02-06T22:17:59.792949+0000","last_clean":"2024-02-06T22:17:59.792949+0000","last_became_active":"2024-02-06T21:42:49.754000+0000","last_became_peered":"2024-02-06T21:42:49.754000+0000","last_unstale":"2024-02-06T22:17:59.792949+0000","last_undegraded":"2024-02-06T22:17:59.792949+0000","last_fullsized":"2024-02-06T22:17:59.792949+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T21:42:37.743292+0000","last_clean_scrub_stamp":"2024-02-06T21:42:37.743292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T06:23:56.755422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":18,"seq":77309411769,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72696,"kb_used_data":5416,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299144,"statfs":{"total":96636764160,"available":96562323456,"internally_reserved":0,"allocated":5545984,"data_stored":2794404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":58,"apply_latency_ms":58,"commit_latency_ns":58000000,"apply_latency_ns":58000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:41 2024","interfaces":[{"interface":"back","average":{"1min":15.664999999999999,"5min":29.364000000000001,"15min":24.5},"min":{"1min":5.9740000000000002,"5min":4.96,"15min":4.5209999999999999},"max":{"1min":76.638000000000005,"5min":223.274,"15min":549.32299999999998},"last":11.345000000000001},{"interface":"front","average":{"1min":20.262,"5min":35.253999999999998,"15min":29.021999999999998},"min":{"1min":10.795999999999999,"5min":7.9969999999999999,"15min":5.4900000000000002},"max":{"1min":78.369,"5min":224.31700000000001,"15min":368.31200000000001},"last":15.666}]},{"osd":2,"last update":"Tue Feb 6 22:17:09 2024","interfaces":[{"interface":"back","average":{"1min":23.908999999999999,"5min":30.920999999999999,"15min":26.542999999999999},"min":{"1min":11.26,"5min":5.1520000000000001,"15min":5.1520000000000001},"max":{"1min":88.763000000000005,"5min":205.28800000000001,"15min":387.75299999999999},"last":16.474},{"interface":"front","average":{"1min":12.706,"5min":29.155000000000001,"15min":31.622},"min":{"1min":5.3460000000000001,"5min":5.3460000000000001,"15min":5.3460000000000001},"max":{"1min":43.438000000000002,"5min":220.31899999999999,"15min":442.25},"last":11.977}]},{"osd":3,"last update":"Tue Feb 6 22:17:55 2024","interfaces":[{"interface":"back","average":{"1min":11.712999999999999,"5min":24.257999999999999,"15min":21.895},"min":{"1min":6.1520000000000001,"5min":5.3899999999999997,"15min":5.3899999999999997},"max":{"1min":68.462999999999994,"5min":359.19299999999998,"15min":359.19299999999998},"last":7.5190000000000001},{"interface":"front","average":{"1min":26.09,"5min":35.533999999999999,"15min":31.292999999999999},"min":{"1min":11.449,"5min":8.5500000000000007,"15min":7.8010000000000002},"max":{"1min":102.22199999999999,"5min":266.40600000000001,"15min":462.86200000000002},"last":13.945}]},{"osd":4,"last update":"Tue Feb 6 22:17:24 2024","interfaces":[{"interface":"back","average":{"1min":25.393999999999998,"5min":39.593000000000004,"15min":29.221},"min":{"1min":12.670999999999999,"5min":9.0559999999999992,"15min":5.9539999999999997},"max":{"1min":85.082999999999998,"5min":535.22500000000002,"15min":535.22500000000002},"last":14.641999999999999},{"interface":"front","average":{"1min":12.597,"5min":28.677,"15min":22.689},"min":{"1min":7.5220000000000002,"5min":6.3010000000000002,"15min":5.4649999999999999},"max":{"1min":49.466000000000001,"5min":473.69799999999998,"15min":473.69799999999998},"last":8.0440000000000005}]},{"osd":5,"last update":"Tue Feb 6 22:17:41 2024","interfaces":[{"interface":"back","average":{"1min":27.221,"5min":36.164999999999999,"15min":32.499000000000002},"min":{"1min":14.074999999999999,"5min":8.0459999999999994,"15min":6.4550000000000001},"max":{"1min":90.605999999999995,"5min":260.74400000000003,"15min":454.49099999999999},"last":17.385999999999999},{"interface":"front","average":{"1min":17.254000000000001,"5min":25.736999999999998,"15min":26.802},"min":{"1min":8.1069999999999993,"5min":6.8010000000000002,"15min":6.109},"max":{"1min":70.180000000000007,"5min":147.011,"15min":539.04700000000003},"last":10.371}]},{"osd":6,"last update":"Tue Feb 6 22:17:24 2024","interfaces":[{"interface":"back","average":{"1min":27.564,"5min":40.116999999999997,"15min":37.177},"min":{"1min":13.347,"5min":12.289,"15min":10.353999999999999},"max":{"1min":89.951999999999998,"5min":268.73399999999998,"15min":482.846},"last":19.059999999999999},{"interface":"front","average":{"1min":16.326000000000001,"5min":27.145,"15min":24.978000000000002},"min":{"1min":8.7759999999999998,"5min":6.9349999999999996,"15min":6.9349999999999996},"max":{"1min":51.412999999999997,"5min":186.95500000000001,"15min":322.64999999999998},"last":12.529}]},{"osd":7,"last update":"Tue Feb 6 22:17:41 2024","interfaces":[{"interface":"back","average":{"1min":15.381,"5min":27.943000000000001,"15min":30.135000000000002},"min":{"1min":6.1669999999999998,"5min":6.1669999999999998,"15min":6.1669999999999998},"max":{"1min":75.899000000000001,"5min":249.69200000000001,"15min":452.22399999999999},"last":9.6549999999999994},{"interface":"front","average":{"1min":31.131,"5min":42.412999999999997,"15min":35.392000000000003},"min":{"1min":13.923999999999999,"5min":10.964,"15min":7.7030000000000003},"max":{"1min":99.406000000000006,"5min":292.94600000000003,"15min":563.26900000000001},"last":18.003}]}]},{"osd":6,"up_from":17,"seq":73014444474,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72300,"kb_used_data":5024,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299540,"statfs":{"total":96636764160,"available":96562728960,"internally_reserved":0,"allocated":5144576,"data_stored":2396564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":14.135,"5min":21.481000000000002,"15min":17.268000000000001},"min":{"1min":4.3339999999999996,"5min":4.3230000000000004,"15min":4.3230000000000004},"max":{"1min":59.670999999999999,"5min":188.761,"15min":218.85400000000001},"last":4.4039999999999999},{"interface":"front","average":{"1min":11.811,"5min":24.353000000000002,"15min":17.314},"min":{"1min":5.1559999999999997,"5min":5.1559999999999997,"15min":4.8449999999999998},"max":{"1min":38.082999999999998,"5min":193.68000000000001,"15min":193.68000000000001},"last":11.744999999999999}]},{"osd":1,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":15.811999999999999,"5min":23.908999999999999,"15min":21.940000000000001},"min":{"1min":6.431,"5min":6.1459999999999999,"15min":5.109},"max":{"1min":54.000999999999998,"5min":180.447,"15min":204.64500000000001},"last":27.359999999999999},{"interface":"front","average":{"1min":17.199000000000002,"5min":23.283000000000001,"15min":24.012},"min":{"1min":6.7949999999999999,"5min":6.7949999999999999,"15min":6.1070000000000002},"max":{"1min":60.319000000000003,"5min":168.404,"15min":460.71600000000001},"last":7.5979999999999999}]},{"osd":2,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":19.161000000000001,"5min":36.042000000000002,"15min":26.562999999999999},"min":{"1min":8.0570000000000004,"5min":6.9359999999999999,"15min":5.5330000000000004},"max":{"1min":72.200000000000003,"5min":794.04200000000003,"15min":794.04200000000003},"last":24.497},{"interface":"front","average":{"1min":19.587,"5min":43.002000000000002,"15min":27.474},"min":{"1min":11.856,"5min":6.2999999999999998,"15min":5.601},"max":{"1min":62.237000000000002,"5min":795.17499999999995,"15min":795.17499999999995},"last":11.057}]},{"osd":3,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":19.741,"5min":32.057000000000002,"15min":24.117000000000001},"min":{"1min":10.901999999999999,"5min":7.718,"15min":5.9240000000000004},"max":{"1min":72.951999999999998,"5min":317.899,"15min":317.899},"last":29.620999999999999},{"interface":"front","average":{"1min":20.227,"5min":29.085999999999999,"15min":21.300999999999998},"min":{"1min":10.553000000000001,"5min":6.468,"15min":6.0380000000000003},"max":{"1min":65.974000000000004,"5min":405.108,"15min":405.108},"last":21.326000000000001}]},{"osd":4,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":10.769,"5min":23.568999999999999,"15min":22.443999999999999},"min":{"1min":5.8940000000000001,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":30.241,"5min":231.55699999999999,"15min":243.667},"last":9.3049999999999997},{"interface":"front","average":{"1min":14.613,"5min":30.960000000000001,"15min":27.713000000000001},"min":{"1min":8.641,"5min":7.7160000000000002,"15min":7.3399999999999999},"max":{"1min":39.484999999999999,"5min":284.37,"15min":339.34399999999999},"last":23.623000000000001}]},{"osd":5,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":20.939,"5min":24.574999999999999,"15min":22.995000000000001},"min":{"1min":11.087999999999999,"5min":9.2929999999999993,"15min":7.5990000000000002},"max":{"1min":67.230000000000004,"5min":157.095,"15min":240.20500000000001},"last":19.751000000000001},{"interface":"front","average":{"1min":23.254999999999999,"5min":31.707999999999998,"15min":27.795000000000002},"min":{"1min":14.375,"5min":13.481,"15min":9.9260000000000002},"max":{"1min":69.085999999999999,"5min":202.673,"15min":223.42500000000001},"last":25.914999999999999}]},{"osd":7,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":23.655999999999999,"5min":30.402999999999999,"15min":23.09},"min":{"1min":15.859,"5min":11.272,"15min":7.4580000000000002},"max":{"1min":70.263999999999996,"5min":190.77799999999999,"15min":224.63},"last":25.323},{"interface":"front","average":{"1min":21.408999999999999,"5min":31.376999999999999,"15min":27.039999999999999},"min":{"1min":8.7430000000000003,"5min":8.7430000000000003,"15min":8.7430000000000003},"max":{"1min":74.259,"5min":203.26499999999999,"15min":220.893},"last":27.968}]}]},{"osd":5,"up_from":17,"seq":73014444476,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72688,"kb_used_data":5412,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299152,"statfs":{"total":96636764160,"available":96562331648,"internally_reserved":0,"allocated":5541888,"data_stored":2794399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":14.396000000000001,"5min":19.771999999999998,"15min":16.498999999999999},"min":{"1min":6.2709999999999999,"5min":5.6180000000000003,"15min":4.5090000000000003},"max":{"1min":72.215999999999994,"5min":115.977,"15min":350.07600000000002},"last":8.0709999999999997},{"interface":"front","average":{"1min":20.567,"5min":26.533999999999999,"15min":22.433},"min":{"1min":9.1340000000000003,"5min":7.8810000000000002,"15min":6.133},"max":{"1min":81.966999999999999,"5min":128.446,"15min":351.94},"last":15}]},{"osd":1,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":16.960000000000001,"5min":22.443000000000001,"15min":21.648},"min":{"1min":7.9630000000000001,"5min":7.0830000000000002,"15min":6.2359999999999998},"max":{"1min":46.289000000000001,"5min":158.76400000000001,"15min":275.12},"last":11.101000000000001},{"interface":"front","average":{"1min":19.393000000000001,"5min":24.099,"15min":22.812999999999999},"min":{"1min":8.5939999999999994,"5min":8.1590000000000007,"15min":4.6520000000000001},"max":{"1min":61.463000000000001,"5min":112.78700000000001,"15min":369.73200000000003},"last":19.456}]},{"osd":2,"last update":"Tue Feb 6 22:17:21 2024","interfaces":[{"interface":"back","average":{"1min":12.752000000000001,"5min":43.362000000000002,"15min":31.550000000000001},"min":{"1min":5.7140000000000004,"5min":5.7140000000000004,"15min":5.7140000000000004},"max":{"1min":46.295000000000002,"5min":2330.5610000000001,"15min":2330.5610000000001},"last":10.586},{"interface":"front","average":{"1min":17.791,"5min":50.844999999999999,"15min":35.262999999999998},"min":{"1min":7.3230000000000004,"5min":7.1390000000000002,"15min":7.1390000000000002},"max":{"1min":44.875999999999998,"5min":2614.4409999999998,"15min":2614.4409999999998},"last":16.094999999999999}]},{"osd":3,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":18.696000000000002,"5min":23.902000000000001,"15min":24.218},"min":{"1min":7.8049999999999997,"5min":6.125,"15min":6.125},"max":{"1min":62.171999999999997,"5min":156.80000000000001,"15min":393.55200000000002},"last":17.385000000000002},{"interface":"front","average":{"1min":14.869,"5min":19.66,"15min":21.309999999999999},"min":{"1min":6.1159999999999997,"5min":5.3819999999999997,"15min":5.3819999999999997},"max":{"1min":63.813000000000002,"5min":104.755,"15min":376.113},"last":8.609}]},{"osd":4,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":21.885000000000002,"15min":23.100999999999999},"min":{"1min":6.5650000000000004,"5min":6.093,"15min":6.093},"max":{"1min":84.843999999999994,"5min":105.532,"15min":426.77300000000002},"last":11.35},{"interface":"front","average":{"1min":18.773,"5min":21.994,"15min":24.087},"min":{"1min":7.4870000000000001,"5min":7.3819999999999997,"15min":5.8330000000000002},"max":{"1min":50.149000000000001,"5min":90.284000000000006,"15min":419.16000000000003},"last":16.757999999999999}]},{"osd":6,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":20.283000000000001,"5min":29.465,"15min":28.431999999999999},"min":{"1min":6.9560000000000004,"5min":6.9560000000000004,"15min":6.9560000000000004},"max":{"1min":85.828000000000003,"5min":219.14099999999999,"15min":392.71600000000001},"last":12.5},{"interface":"front","average":{"1min":21.872,"5min":25.550000000000001,"15min":26.341000000000001},"min":{"1min":9.8740000000000006,"5min":7.2350000000000003,"15min":7.2350000000000003},"max":{"1min":65.367000000000004,"5min":152.69900000000001,"15min":377.31400000000002},"last":17.995999999999999}]},{"osd":7,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":19.959,"5min":25.904,"15min":25.827999999999999},"min":{"1min":9.0510000000000002,"5min":9.0510000000000002,"15min":9.0510000000000002},"max":{"1min":63.646000000000001,"5min":124.711,"15min":377.87400000000002},"last":13.358000000000001},{"interface":"front","average":{"1min":19.725000000000001,"5min":27.969999999999999,"15min":24.045999999999999},"min":{"1min":7.8940000000000001,"5min":7.8940000000000001,"15min":6.7549999999999999},"max":{"1min":48.325000000000003,"5min":152.59399999999999,"15min":338.803},"last":14.266999999999999}]}]},{"osd":4,"up_from":17,"seq":73014444473,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72296,"kb_used_data":5020,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299544,"statfs":{"total":96636764160,"available":96562733056,"internally_reserved":0,"allocated":5140480,"data_stored":2396559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.2439999999999998,"5min":17.853000000000002,"15min":16.888999999999999},"min":{"1min":4.907,"5min":4.4169999999999998,"15min":4.3899999999999997},"max":{"1min":19.225999999999999,"5min":167.291,"15min":216.86600000000001},"last":17.149999999999999},{"interface":"front","average":{"1min":9.9450000000000003,"5min":22.707999999999998,"15min":18.631},"min":{"1min":4.742,"5min":4.6920000000000002,"15min":4.6829999999999998},"max":{"1min":23.745000000000001,"5min":231.82599999999999,"15min":287.97300000000001},"last":14.898999999999999}]},{"osd":1,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":17.431999999999999,"5min":24.254999999999999,"15min":30.890000000000001},"min":{"1min":6.2949999999999999,"5min":6.2949999999999999,"15min":5.1539999999999999},"max":{"1min":65.745999999999995,"5min":207.22399999999999,"15min":678.16999999999996},"last":11.593999999999999},{"interface":"front","average":{"1min":15.343999999999999,"5min":23.465,"15min":32.118000000000002},"min":{"1min":8.4299999999999997,"5min":6.2599999999999998,"15min":6.2599999999999998},"max":{"1min":25.268999999999998,"5min":203.19999999999999,"15min":667.16300000000001},"last":9.4730000000000008}]},{"osd":2,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":15.932,"5min":59.622999999999998,"15min":35.741999999999997},"min":{"1min":10.554,"5min":6.601,"15min":6.601},"max":{"1min":32.744999999999997,"5min":2277.7020000000002,"15min":2277.7020000000002},"last":13.012},{"interface":"front","average":{"1min":22.949999999999999,"5min":66.596999999999994,"15min":44.031999999999996},"min":{"1min":12.722,"5min":9.4559999999999995,"15min":9.4559999999999995},"max":{"1min":52.287999999999997,"5min":2279.0419999999999,"15min":2279.0419999999999},"last":13.622999999999999}]},{"osd":3,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":21.295999999999999,"5min":30.468,"15min":28.460999999999999},"min":{"1min":9.4499999999999993,"5min":6.3559999999999999,"15min":6.3559999999999999},"max":{"1min":62.805999999999997,"5min":254.292,"15min":423.20400000000001},"last":16.372},{"interface":"front","average":{"1min":15.101000000000001,"5min":31.655000000000001,"15min":24.419},"min":{"1min":6.9610000000000003,"5min":6.9610000000000003,"15min":5.5910000000000002},"max":{"1min":25.286999999999999,"5min":225.64699999999999,"15min":359.959},"last":10.83}]},{"osd":5,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":22.587,"5min":34.43,"15min":29.388999999999999},"min":{"1min":12.337999999999999,"5min":10.756,"15min":7.2999999999999998},"max":{"1min":66.715999999999994,"5min":248.70500000000001,"15min":363.36900000000003},"last":17.861999999999998},{"interface":"front","average":{"1min":15.039999999999999,"5min":28.181999999999999,"15min":27.018999999999998},"min":{"1min":6.6040000000000001,"5min":6.476,"15min":6.1550000000000002},"max":{"1min":43.223999999999997,"5min":208.02699999999999,"15min":809.58100000000002},"last":8.0449999999999999}]},{"osd":6,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":17.777999999999999,"5min":24.445,"15min":26.818999999999999},"min":{"1min":7.5819999999999999,"5min":6.2930000000000001,"15min":6.2930000000000001},"max":{"1min":63.929000000000002,"5min":174.40700000000001,"15min":400.46800000000002},"last":6.9820000000000002},{"interface":"front","average":{"1min":24.074999999999999,"5min":35.155000000000001,"15min":36.561},"min":{"1min":13.917,"5min":11.411,"15min":10.502000000000001},"max":{"1min":66.519999999999996,"5min":269.44200000000001,"15min":498.22300000000001},"last":14.257999999999999}]},{"osd":7,"last update":"Tue Feb 6 22:17:04 2024","interfaces":[{"interface":"back","average":{"1min":24.492000000000001,"5min":35.420000000000002,"15min":26.943000000000001},"min":{"1min":14.81,"5min":8.9009999999999998,"15min":6.6239999999999997},"max":{"1min":52.576999999999998,"5min":287.61599999999999,"15min":360.55599999999998},"last":18.998999999999999},{"interface":"front","average":{"1min":15.82,"5min":26.690999999999999,"15min":28.559999999999999},"min":{"1min":9.7210000000000001,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":28.702999999999999,"5min":215.26400000000001,"15min":478.649},"last":10.138}]}]},{"osd":3,"up_from":17,"seq":73014444473,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72296,"kb_used_data":5020,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299544,"statfs":{"total":96636764160,"available":96562733056,"internally_reserved":0,"allocated":5140480,"data_stored":2396559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:35 2024","interfaces":[{"interface":"back","average":{"1min":15.148999999999999,"5min":19.969999999999999,"15min":18.234000000000002},"min":{"1min":5.6100000000000003,"5min":4.5270000000000001,"15min":4.173},"max":{"1min":37.642000000000003,"5min":178.14400000000001,"15min":233.84200000000001},"last":7.4889999999999999},{"interface":"front","average":{"1min":17.673999999999999,"5min":22.867000000000001,"15min":22.109999999999999},"min":{"1min":7.1299999999999999,"5min":5.3949999999999996,"15min":5.3949999999999996},"max":{"1min":45.851999999999997,"5min":224.822,"15min":224.822},"last":16.324999999999999}]},{"osd":1,"last update":"Tue Feb 6 22:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.183999999999999,"5min":19.210000000000001,"15min":21.574000000000002},"min":{"1min":6.7039999999999997,"5min":6.7039999999999997,"15min":6.7039999999999997},"max":{"1min":21.882999999999999,"5min":169.31100000000001,"15min":273.12700000000001},"last":19.084},{"interface":"front","average":{"1min":16.084,"5min":25.670000000000002,"15min":25.431999999999999},"min":{"1min":10.845000000000001,"5min":10.845000000000001,"15min":9.2880000000000003},"max":{"1min":24.265000000000001,"5min":205.495,"15min":322.70499999999998},"last":18.239999999999998}]},{"osd":2,"last update":"Tue Feb 6 22:17:35 2024","interfaces":[{"interface":"back","average":{"1min":25.116,"5min":28.814,"15min":26.242999999999999},"min":{"1min":11.129,"5min":11.129,"15min":9.1999999999999993},"max":{"1min":59.799999999999997,"5min":240.316,"15min":282.40699999999998},"last":17},{"interface":"front","average":{"1min":11.965,"5min":28.228999999999999,"15min":21.456},"min":{"1min":5.8979999999999997,"5min":5.8419999999999996,"15min":5.1550000000000002},"max":{"1min":39.423000000000002,"5min":998.23500000000001,"15min":998.23500000000001},"last":13.130000000000001}]},{"osd":4,"last update":"Tue Feb 6 22:17:17 2024","interfaces":[{"interface":"back","average":{"1min":12.571999999999999,"5min":17.777000000000001,"15min":20.849},"min":{"1min":7.6459999999999999,"5min":6.1109999999999998,"15min":5.8339999999999996},"max":{"1min":25.420000000000002,"5min":162.65299999999999,"15min":189.12},"last":14.692},{"interface":"front","average":{"1min":18.823,"5min":27.978999999999999,"15min":31.401},"min":{"1min":12.393000000000001,"5min":11.710000000000001,"15min":11.045},"max":{"1min":30.169,"5min":176.35400000000001,"15min":235.315},"last":19.923999999999999}]},{"osd":5,"last update":"Tue Feb 6 22:17:35 2024","interfaces":[{"interface":"back","average":{"1min":29.649000000000001,"5min":32.82,"15min":29.718},"min":{"1min":10.526,"5min":10.087999999999999,"15min":8.9920000000000009},"max":{"1min":77.253,"5min":264.06999999999999,"15min":626.27700000000004},"last":20.547999999999998},{"interface":"front","average":{"1min":37.348999999999997,"5min":37.712000000000003,"15min":32.811999999999998},"min":{"1min":13.699,"5min":13.699,"15min":13.699},"max":{"1min":112.84699999999999,"5min":284.88299999999998,"15min":614.85400000000004},"last":22.364000000000001}]},{"osd":6,"last update":"Tue Feb 6 22:17:54 2024","interfaces":[{"interface":"back","average":{"1min":33.225999999999999,"5min":29.445,"15min":22.18},"min":{"1min":16.498000000000001,"5min":6.6230000000000002,"15min":6.218},"max":{"1min":75.498000000000005,"5min":177.40700000000001,"15min":234.43700000000001},"last":21.571999999999999},{"interface":"front","average":{"1min":19.661999999999999,"5min":30.109000000000002,"15min":28.652000000000001},"min":{"1min":7.258,"5min":7.258,"15min":7.258},"max":{"1min":66.543999999999997,"5min":209.25800000000001,"15min":232.30000000000001},"last":10.401}]},{"osd":7,"last update":"Tue Feb 6 22:17:54 2024","interfaces":[{"interface":"back","average":{"1min":20.655000000000001,"5min":23.515000000000001,"15min":22.335999999999999},"min":{"1min":9.3759999999999994,"5min":7.3579999999999997,"15min":7.3579999999999997},"max":{"1min":75.813999999999993,"5min":173.642,"15min":235.012},"last":11.82},{"interface":"front","average":{"1min":36.191000000000003,"5min":34.972999999999999,"15min":31.428999999999998},"min":{"1min":17.329000000000001,"5min":14.802,"15min":12.015000000000001},"max":{"1min":117.54600000000001,"5min":206.87,"15min":233.11199999999999},"last":23.198}]}]},{"osd":2,"up_from":17,"seq":73014444473,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72296,"kb_used_data":5020,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299544,"statfs":{"total":96636764160,"available":96562733056,"internally_reserved":0,"allocated":5140480,"data_stored":2396559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:58 2024","interfaces":[{"interface":"back","average":{"1min":16.789000000000001,"5min":41.939,"15min":24.574999999999999},"min":{"1min":4.0780000000000003,"5min":4.0780000000000003,"15min":4.0780000000000003},"max":{"1min":128.39599999999999,"5min":1631.963,"15min":1631.963},"last":10.957000000000001},{"interface":"front","average":{"1min":14.339,"5min":22.324000000000002,"15min":20.181999999999999},"min":{"1min":6.1740000000000004,"5min":4.4859999999999998,"15min":3.9630000000000001},"max":{"1min":109.867,"5min":138.09800000000001,"15min":370.68799999999999},"last":10.65}]},{"osd":1,"last update":"Tue Feb 6 22:17:26 2024","interfaces":[{"interface":"back","average":{"1min":10.836,"5min":52.090000000000003,"15min":32.5},"min":{"1min":6.9000000000000004,"5min":5.9340000000000002,"15min":4.492},"max":{"1min":38.810000000000002,"5min":3146.2530000000002,"15min":3146.2530000000002},"last":7.5389999999999997},{"interface":"front","average":{"1min":19.111000000000001,"5min":58.341999999999999,"15min":36.834000000000003},"min":{"1min":8.4320000000000004,"5min":6.9669999999999996,"15min":6.9020000000000001},"max":{"1min":111.504,"5min":3147.6979999999999,"15min":3147.6979999999999},"last":9.6120000000000001}]},{"osd":3,"last update":"Tue Feb 6 22:17:58 2024","interfaces":[{"interface":"back","average":{"1min":12.185,"5min":51.636000000000003,"15min":30.648},"min":{"1min":7.7939999999999996,"5min":7.7939999999999996,"15min":6.1239999999999997},"max":{"1min":18.716999999999999,"5min":2357.8649999999998,"15min":2357.8649999999998},"last":12.726000000000001},{"interface":"front","average":{"1min":15.359,"5min":58.621000000000002,"15min":35.381999999999998},"min":{"1min":10.959,"5min":9.6289999999999996,"15min":7.133},"max":{"1min":21.091000000000001,"5min":2367.828,"15min":2367.828},"last":15.805}]},{"osd":4,"last update":"Tue Feb 6 22:17:58 2024","interfaces":[{"interface":"back","average":{"1min":11.063000000000001,"5min":44.146000000000001,"15min":33.569000000000003},"min":{"1min":7.6319999999999997,"5min":7.5949999999999998,"15min":6.5129999999999999},"max":{"1min":15.863,"5min":1805.6099999999999,"15min":1805.6099999999999},"last":11.968999999999999},{"interface":"front","average":{"1min":15.403,"5min":23.297999999999998,"15min":24.088999999999999},"min":{"1min":9.2959999999999994,"5min":6.976,"15min":6.4009999999999998},"max":{"1min":21.530999999999999,"5min":139.29900000000001,"15min":366.25099999999998},"last":13.589}]},{"osd":5,"last update":"Tue Feb 6 22:17:58 2024","interfaces":[{"interface":"back","average":{"1min":22.925999999999998,"5min":61.741,"15min":37.472999999999999},"min":{"1min":10.340999999999999,"5min":7.0650000000000004,"15min":6.9610000000000003},"max":{"1min":119.056,"5min":3145.4110000000001,"15min":3145.4110000000001},"last":14.452},{"interface":"front","average":{"1min":22.033000000000001,"5min":59.262999999999998,"15min":39.688000000000002},"min":{"1min":12.734,"5min":10.861000000000001,"15min":8.3140000000000001},"max":{"1min":53.290999999999997,"5min":2302.047,"15min":2302.047},"last":17.07}]},{"osd":6,"last update":"Tue Feb 6 22:17:58 2024","interfaces":[{"interface":"back","average":{"1min":15.878,"5min":31.748000000000001,"15min":28.632000000000001},"min":{"1min":10.866,"5min":7.7039999999999997,"15min":6.6479999999999997},"max":{"1min":24.971,"5min":266.53500000000003,"15min":369.185},"last":15.26},{"interface":"front","average":{"1min":19.640999999999998,"5min":60.502000000000002,"15min":39.997999999999998},"min":{"1min":13.670999999999999,"5min":12.715,"15min":8.4930000000000003},"max":{"1min":25.183,"5min":2302.8209999999999,"15min":2302.8209999999999},"last":18.661999999999999}]},{"osd":7,"last update":"Tue Feb 6 22:17:58 2024","interfaces":[{"interface":"back","average":{"1min":18.395,"5min":28.178000000000001,"15min":24.724},"min":{"1min":9.3499999999999996,"5min":8.7119999999999997,"15min":8.4619999999999997},"max":{"1min":69.673000000000002,"5min":267.28300000000002,"15min":335.39499999999998},"last":17.91},{"interface":"front","average":{"1min":25.317,"5min":33.137,"15min":28.495000000000001},"min":{"1min":13.324,"5min":11.890000000000001,"15min":10.06},"max":{"1min":151.32499999999999,"5min":268.60899999999998,"15min":342.22699999999998},"last":19.678999999999998}]}]},{"osd":7,"up_from":13,"seq":55834575288,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76456,"kb_used_data":5020,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295384,"statfs":{"total":96636764160,"available":96558473216,"internally_reserved":0,"allocated":5140480,"data_stored":2396559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":11.686999999999999,"5min":23.573,"15min":18.436},"min":{"1min":5.0789999999999997,"5min":4.2880000000000003,"15min":4.2880000000000003},"max":{"1min":28.937000000000001,"5min":283.42899999999997,"15min":283.42899999999997},"last":52.298999999999999},{"interface":"front","average":{"1min":12.000999999999999,"5min":23.219999999999999,"15min":19.027999999999999},"min":{"1min":5.2949999999999999,"5min":4.3730000000000002,"15min":4.3730000000000002},"max":{"1min":33.189,"5min":279.459,"15min":387.32999999999998},"last":51.878999999999998}]},{"osd":1,"last update":"Tue Feb 6 22:17:29 2024","interfaces":[{"interface":"back","average":{"1min":13.608000000000001,"5min":19.052,"15min":22.741},"min":{"1min":6.0609999999999999,"5min":6.0609999999999999,"15min":5.577},"max":{"1min":31.280999999999999,"5min":145.70599999999999,"15min":325.41699999999997},"last":12.286},{"interface":"front","average":{"1min":13.199999999999999,"5min":17.952000000000002,"15min":21.225999999999999},"min":{"1min":6.2720000000000002,"5min":6.2720000000000002,"15min":4.7960000000000003},"max":{"1min":30.079999999999998,"5min":159.56100000000001,"15min":380.70800000000003},"last":14.372999999999999}]},{"osd":2,"last update":"Tue Feb 6 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":15.321999999999999,"5min":78.852000000000004,"15min":42.732999999999997},"min":{"1min":7.1890000000000001,"5min":6.367,"15min":5.8769999999999998},"max":{"1min":56.795000000000002,"5min":3537.9119999999998,"15min":3537.9119999999998},"last":13.648999999999999},{"interface":"front","average":{"1min":17,"5min":81.25,"15min":44.497},"min":{"1min":9.3260000000000005,"5min":7.6790000000000003,"15min":6.6539999999999999},"max":{"1min":26.536999999999999,"5min":3538.9259999999999,"15min":3538.9259999999999},"last":16.547999999999998}]},{"osd":3,"last update":"Tue Feb 6 22:17:29 2024","interfaces":[{"interface":"back","average":{"1min":16.347999999999999,"5min":20.353999999999999,"15min":21.109999999999999},"min":{"1min":7.7990000000000004,"5min":6.8399999999999999,"15min":6.3390000000000004},"max":{"1min":57.917999999999999,"5min":186.297,"15min":317.94600000000003},"last":15.074999999999999},{"interface":"front","average":{"1min":22.106000000000002,"5min":27.684000000000001,"15min":27.027999999999999},"min":{"1min":14.65,"5min":9.8360000000000003,"15min":9.8360000000000003},"max":{"1min":58.795000000000002,"5min":201.75299999999999,"15min":325.60599999999999},"last":19.556999999999999}]},{"osd":4,"last update":"Tue Feb 6 22:17:20 2024","interfaces":[{"interface":"back","average":{"1min":20.239999999999998,"5min":22.088999999999999,"15min":22.254999999999999},"min":{"1min":9.7159999999999993,"5min":9.2370000000000001,"15min":6.9690000000000003},"max":{"1min":61.972999999999999,"5min":102.001,"15min":235.27799999999999},"last":18.745999999999999},{"interface":"front","average":{"1min":14.773999999999999,"5min":18.849,"15min":25.335000000000001},"min":{"1min":7.585,"5min":6.8440000000000003,"15min":6.8440000000000003},"max":{"1min":25.353000000000002,"5min":87.968999999999994,"15min":350.33300000000003},"last":13.006}]},{"osd":5,"last update":"Tue Feb 6 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":24.497,"5min":25.561,"15min":24.407},"min":{"1min":12.058,"5min":7.1189999999999998,"15min":7.1189999999999998},"max":{"1min":89.010999999999996,"5min":201.70400000000001,"15min":214.83500000000001},"last":20.370999999999999},{"interface":"front","average":{"1min":18.481999999999999,"5min":22.195,"15min":25.890999999999998},"min":{"1min":8.6969999999999992,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":65.813999999999993,"5min":106.07299999999999,"15min":301.78399999999999},"last":8.5419999999999998}]},{"osd":6,"last update":"Tue Feb 6 22:17:29 2024","interfaces":[{"interface":"back","average":{"1min":23.632999999999999,"5min":26.312999999999999,"15min":25.606000000000002},"min":{"1min":15.241,"5min":8.3239999999999998,"15min":8.3239999999999998},"max":{"1min":60.616,"5min":203.58199999999999,"15min":235.27799999999999},"last":21.451000000000001},{"interface":"front","average":{"1min":16.300999999999998,"5min":22.77,"15min":26.331},"min":{"1min":9.3000000000000007,"5min":7.3179999999999996,"15min":6.7400000000000002},"max":{"1min":28.381,"5min":185.49799999999999,"15min":490.68299999999999},"last":15.824}]}]},{"osd":0,"up_from":13,"seq":55834575285,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72296,"kb_used_data":5020,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299544,"statfs":{"total":96636764160,"available":96562733056,"internally_reserved":0,"allocated":5140480,"data_stored":2396559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 6 22:17:53 2024","interfaces":[{"interface":"back","average":{"1min":15.26,"5min":28.606999999999999,"15min":23.428000000000001},"min":{"1min":5.6970000000000001,"5min":4.633,"15min":4.4029999999999996},"max":{"1min":98.292000000000002,"5min":314.11000000000001,"15min":416.55599999999998},"last":8.2439999999999998},{"interface":"front","average":{"1min":13.664999999999999,"5min":22.065999999999999,"15min":21.591000000000001},"min":{"1min":6.5339999999999998,"5min":5.5389999999999997,"15min":4.5},"max":{"1min":81.653999999999996,"5min":199.238,"15min":407.44799999999998},"last":8.5289999999999999}]},{"osd":2,"last update":"Tue Feb 6 22:17:53 2024","interfaces":[{"interface":"back","average":{"1min":17.803999999999998,"5min":28.433,"15min":27.036999999999999},"min":{"1min":7.2770000000000001,"5min":6.4880000000000004,"15min":6.3799999999999999},"max":{"1min":93.007999999999996,"5min":264.48200000000003,"15min":395.14800000000002},"last":9.173},{"interface":"front","average":{"1min":15.856,"5min":29.306000000000001,"15min":25.323},"min":{"1min":5.9100000000000001,"5min":5.5259999999999998,"15min":5.383},"max":{"1min":99.483000000000004,"5min":514.94500000000005,"15min":514.94500000000005},"last":11.497999999999999}]},{"osd":3,"last update":"Tue Feb 6 22:17:53 2024","interfaces":[{"interface":"back","average":{"1min":19.762,"5min":26.550000000000001,"15min":24.818000000000001},"min":{"1min":8.6289999999999996,"5min":7.2409999999999997,"15min":5.8140000000000001},"max":{"1min":105.68000000000001,"5min":291.59100000000001,"15min":376.95100000000002},"last":9.7989999999999995},{"interface":"front","average":{"1min":14.304,"5min":25.513000000000002,"15min":25.059999999999999},"min":{"1min":7.75,"5min":6.6139999999999999,"15min":6.6139999999999999},"max":{"1min":80.340000000000003,"5min":262.06900000000002,"15min":395.83100000000002},"last":12.709}]},{"osd":4,"last update":"Tue Feb 6 22:17:09 2024","interfaces":[{"interface":"back","average":{"1min":16.298999999999999,"5min":31.213000000000001,"15min":28.899000000000001},"min":{"1min":9.8650000000000002,"5min":7.3609999999999998,"15min":6.6970000000000001},"max":{"1min":27.573,"5min":491.53100000000001,"15min":491.53100000000001},"last":15.128},{"interface":"front","average":{"1min":11.919,"5min":26.989999999999998,"15min":25.725000000000001},"min":{"1min":8.4060000000000006,"5min":7.2409999999999997,"15min":6.6589999999999998},"max":{"1min":18.683,"5min":490.28699999999998,"15min":490.28699999999998},"last":11.081}]},{"osd":5,"last update":"Tue Feb 6 22:17:53 2024","interfaces":[{"interface":"back","average":{"1min":22.463999999999999,"5min":31.809999999999999,"15min":28.338000000000001},"min":{"1min":10.941000000000001,"5min":7.9969999999999999,"15min":7.9969999999999999},"max":{"1min":104.276,"5min":282.00799999999998,"15min":480.24200000000002},"last":16.09},{"interface":"front","average":{"1min":22.041,"5min":27.257000000000001,"15min":23.969000000000001},"min":{"1min":9.7750000000000004,"5min":8.9350000000000005,"15min":7.6059999999999999},"max":{"1min":94.417000000000002,"5min":193.21700000000001,"15min":347.78699999999998},"last":13.667999999999999}]},{"osd":6,"last update":"Tue Feb 6 22:17:53 2024","interfaces":[{"interface":"back","average":{"1min":24.131,"5min":32.994999999999997,"15min":30.001999999999999},"min":{"1min":12.452,"5min":9.4789999999999992,"15min":8.9139999999999997},"max":{"1min":106.277,"5min":218.12,"15min":550.89700000000005},"last":17.77},{"interface":"front","average":{"1min":20.716000000000001,"5min":27.524999999999999,"15min":25.503},"min":{"1min":10.202,"5min":9.1310000000000002,"15min":8.4730000000000008},"max":{"1min":96.307000000000002,"5min":175.947,"15min":486.25299999999999},"last":16.704000000000001}]},{"osd":7,"last update":"Tue Feb 6 22:17:38 2024","interfaces":[{"interface":"back","average":{"1min":24.734000000000002,"5min":33.390999999999998,"15min":30.097000000000001},"min":{"1min":13.772,"5min":11.374000000000001,"15min":9.6170000000000009},"max":{"1min":107.01000000000001,"5min":218.108,"15min":404.22800000000001},"last":14.494999999999999},{"interface":"front","average":{"1min":28.641999999999999,"5min":38.777999999999999,"15min":34.070999999999998},"min":{"1min":15.483000000000001,"5min":13.800000000000001,"15min":11.497999999999999},"max":{"1min":109.504,"5min":219.56299999999999,"15min":415.92000000000002},"last":18.628}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-06T22:18:08.063 INFO:tasks.ceph:pgid 12.1c last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.063 INFO:tasks.ceph:pgid 12.1d last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.063 INFO:tasks.ceph:pgid 12.1f last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.063 INFO:tasks.ceph:pgid 12.0 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.063 INFO:tasks.ceph:pgid 12.1 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.063 INFO:tasks.ceph:pgid 12.2 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.063 INFO:tasks.ceph:pgid 12.3 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.063 INFO:tasks.ceph:pgid 12.4 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.064 INFO:tasks.ceph:pgid 12.5 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.064 INFO:tasks.ceph:pgid 12.6 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.064 INFO:tasks.ceph:pgid 12.7 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.064 INFO:tasks.ceph:pgid 12.d last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.064 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-02-06T21:41:34.053422+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=41, tm_sec=34, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.064 INFO:tasks.ceph:pgid 12.e last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.064 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-02-06T21:41:34.053422+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=41, tm_sec=34, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.064 INFO:tasks.ceph:pgid 12.f last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.064 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-02-06T21:41:34.053422+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=41, tm_sec=34, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.065 INFO:tasks.ceph:pgid 12.c last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.065 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-02-06T21:41:34.053422+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=41, tm_sec=34, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.065 INFO:tasks.ceph:pgid 12.a last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.065 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-02-06T21:41:34.053422+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=41, tm_sec=34, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.065 INFO:tasks.ceph:pgid 12.b last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.065 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-02-06T21:41:34.053422+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=41, tm_sec=34, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.065 INFO:tasks.ceph:pgid 12.8 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.065 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-02-06T21:41:34.053422+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=41, tm_sec=34, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.065 INFO:tasks.ceph:pgid 12.9 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.065 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-02-06T21:41:34.053422+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=41, tm_sec=34, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.066 INFO:tasks.ceph:pgid 12.1b last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.066 INFO:tasks.ceph:pgid 12.1a last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.066 INFO:tasks.ceph:pgid 12.19 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.066 INFO:tasks.ceph:pgid 12.18 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.066 INFO:tasks.ceph:pgid 12.17 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.066 INFO:tasks.ceph:pgid 12.16 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.066 INFO:tasks.ceph:pgid 12.15 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.066 INFO:tasks.ceph:pgid 12.14 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.066 INFO:tasks.ceph:pgid 12.13 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.066 INFO:tasks.ceph:pgid 12.12 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.067 INFO:tasks.ceph:pgid 12.11 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.067 INFO:tasks.ceph:pgid 12.10 last_scrub_stamp 2024-02-06T21:42:37.743292+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=21, tm_min=42, tm_sec=37, tm_wday=1, tm_yday=37, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=6, tm_hour=22, tm_min=17, tm_sec=59, tm_wday=1, tm_yday=37, tm_isdst=0) 2024-02-06T22:18:08.067 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-02-06T22:18:18.843 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:58.658 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:18.847 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:58.662 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:18.867 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:37:58.681 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:18.897 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:58.712 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:18.898 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:58.662 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:18.900 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:58.664 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:18.900 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:58.715 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:18.920 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:37:58.684 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:18.922 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:37:58.737 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:23.851 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:03.666 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:23.856 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:03.671 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:23.867 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:03.682 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:23.903 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:03.718 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:23.904 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:03.668 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:23.905 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:03.720 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:23.908 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:03.672 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:23.915 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:03.730 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:23.918 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:03.683 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:28.068 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-06T22:18:28.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.166+0000 7f601156b640 10 monclient: get_monmap_and_config 2024-02-06T22:18:28.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.167+0000 7f601156b640 10 monclient: build_initial_monmap 2024-02-06T22:18:28.168 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.167+0000 7f601156b640 10 monclient: monmap: 2024-02-06T22:18:28.168 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:28.168 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:28.168 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:28.168282+0000 2024-02-06T22:18:28.169 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:28.168282+0000 2024-02-06T22:18:28.169 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:28.169 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:28.169 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:28.169 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:28.169 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:28.169 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:28.169 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:28.169 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:28.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.167+0000 7f601156b640 1 Processor -- start 2024-02-06T22:18:28.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.167+0000 7f601156b640 1 -- start start 2024-02-06T22:18:28.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.167+0000 7f601156b640 10 monclient: init 2024-02-06T22:18:28.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f601156b640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:28.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f601156b640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-02-06T22:18:28.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f601156b640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f600c0a9c80 0x7f600c0aa060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:28.171 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f601156b640 10 monclient(hunting): picked mon.noname-c con 0x7f600c0a9c80 addr v2:172.21.15.135:3300/0 2024-02-06T22:18:28.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f601156b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f600c142bb0 0x7f600c142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:28.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f601156b640 10 monclient(hunting): picked mon.noname-b con 0x7f600c142bb0 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:28.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f601156b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f600c13f810 addr v1:172.21.15.120:6789/0 2024-02-06T22:18:28.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f601156b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c0aa5a0 con 0x7f600c13f810 2024-02-06T22:18:28.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f601156b640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:28.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f601156b640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f600c143510 con 0x7f600c0a9c80 2024-02-06T22:18:28.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f601156b640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:28.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f601156b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600c142720 con 0x7f600c142bb0 2024-02-06T22:18:28.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f601156b640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:28.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f601156b640 10 monclient(hunting): authenticate will time out at 3538.505808s 2024-02-06T22:18:28.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f600b7fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f600c13f810 0x7f600c13fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:38630/0 (socket says 172.21.15.120:38630) 2024-02-06T22:18:28.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f600b7fe640 1 -- 172.21.15.120:0/242572 learned_addr learned my addr 172.21.15.120:0/242572 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-02-06T22:18:28.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.168+0000 7f600a7fc640 1 --2- 172.21.15.120:0/242572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f600c142bb0 0x7f600c142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:28.172 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.169+0000 7f600affd640 1 --2- 172.21.15.120:0/242572 >> v2:172.21.15.135:3300/0 conn(0x7f600c0a9c80 0x7f600c0aa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:28.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.169+0000 7f600a7fc640 10 monclient(hunting): get_auth_request con 0x7f600c142bb0 auth_method 0 2024-02-06T22:18:28.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.169+0000 7f600a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:28.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.169+0000 7f600a7fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:28.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.169+0000 7f600a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:28.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.169+0000 7f600affd640 10 monclient(hunting): get_auth_request con 0x7f600c0a9c80 auth_method 0 2024-02-06T22:18:28.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.169+0000 7f600affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:28.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.169+0000 7f600affd640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:28.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.169+0000 7f600affd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:28.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.169+0000 7f600a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:28.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.169+0000 7f600a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:28.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.169+0000 7f600a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:28.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.170+0000 7f600affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:28.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.170+0000 7f600affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:28.173 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.170+0000 7f600affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:28.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.171+0000 7f6009ffb640 1 -- 172.21.15.120:0/242572 <== mon.0 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785967953 0 0) 0x7f6004001490 con 0x7f600c13f810 2024-02-06T22:18:28.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.171+0000 7f6009ffb640 10 monclient(hunting): _negotiate 2024-02-06T22:18:28.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.171+0000 7f6009ffb640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:28.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.171+0000 7f6009ffb640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:28.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.171+0000 7f6009ffb640 10 monclient(hunting): my global_id is 16602 2024-02-06T22:18:28.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.171+0000 7f6009ffb640 1 -- 172.21.15.120:0/242572 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff4003520 con 0x7f600c13f810 2024-02-06T22:18:28.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.171+0000 7f600a7fc640 10 monclient(hunting): handle_auth_done global_id 18761 payload 274 2024-02-06T22:18:28.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.171+0000 7f600a7fc640 1 -- 172.21.15.120:0/242572 >> v2:172.21.15.135:3300/0 conn(0x7f600c0a9c80 msgr2=0x7f600c0aa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:28.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.171+0000 7f600a7fc640 1 --2- 172.21.15.120:0/242572 >> v2:172.21.15.135:3300/0 conn(0x7f600c0a9c80 0x7f600c0aa060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.171+0000 7f600a7fc640 1 -- 172.21.15.120:0/242572 >> v1:172.21.15.120:6789/0 conn(0x7f600c13f810 legacy=0x7f600c13fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:28.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.171+0000 7f600a7fc640 10 monclient: _finish_hunting 0 2024-02-06T22:18:28.175 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.171+0000 7f600a7fc640 1 monclient: found mon.noname-b 2024-02-06T22:18:28.176 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.171+0000 7f600a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:28.176 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.171+0000 7f600a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.120:3301/0 2024-02-06T22:18:28.176 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.171+0000 7f600a7fc640 1 -- 172.21.15.120:0/242572 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c0ab120 con 0x7f600c142bb0 2024-02-06T22:18:28.176 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.173+0000 7f600a7fc640 1 --2- 172.21.15.120:0/242572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f600c142bb0 0x7f600c142fd0 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f60000097a0 tx=0x7f600002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2655e68409cad8c7 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:28.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f6009ffb640 1 -- 172.21.15.120:0/242572 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6000031830 con 0x7f600c142bb0 2024-02-06T22:18:28.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f6009ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:28.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f6009ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-02-06T22:18:28.178 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f6009ffb640 10 monclient: dump: 2024-02-06T22:18:28.178 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:28.178 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:28.178 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.178 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.179 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:28.179 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:28.179 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:28.179 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:28.179 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:28.179 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:28.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f6009ffb640 10 monclient: _finish_auth 0 2024-02-06T22:18:28.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f6009ffb640 10 monclient: _check_auth_tickets 2024-02-06T22:18:28.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f6009ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:28.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f6009ffb640 1 -- 172.21.15.120:0/242572 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6000031990 con 0x7f600c142bb0 2024-02-06T22:18:28.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f601156b640 5 monclient: authenticate success, global_id 18761 2024-02-06T22:18:28.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f601156b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-02-06T22:18:28.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f6009ffb640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:28.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f6009ffb640 1 -- 172.21.15.120:0/242572 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6000031d20 con 0x7f600c142bb0 2024-02-06T22:18:28.179 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f601156b640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:28.180 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f601156b640 4 set_mon_vals no callback set 2024-02-06T22:18:28.180 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f6009ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:28.180 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f6009ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-02-06T22:18:28.180 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.176+0000 7f6009ffb640 10 monclient: dump: 2024-02-06T22:18:28.180 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:28.180 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:28.180 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.180 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.180 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:28.180 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:28.180 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:28.180 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:28.180 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:28.180 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:28.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.177+0000 7f601156b640 10 monclient: shutdown 2024-02-06T22:18:28.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.177+0000 7f601156b640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:28.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.177+0000 7f601156b640 1 -- 172.21.15.120:0/242572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f600c142bb0 msgr2=0x7f600c142fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:28.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.177+0000 7f601156b640 1 --2- 172.21.15.120:0/242572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f600c142bb0 0x7f600c142fd0 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f60000097a0 tx=0x7f600002b8d0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.177+0000 7f601156b640 1 -- 172.21.15.120:0/242572 shutdown_connections 2024-02-06T22:18:28.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.177+0000 7f601156b640 1 --2- 172.21.15.120:0/242572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f600c142bb0 0x7f600c142fd0 unknown :-1 s=CLOSED pgs=3354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.177+0000 7f601156b640 1 --2- 172.21.15.120:0/242572 >> v2:172.21.15.135:3300/0 conn(0x7f600c0a9c80 0x7f600c0aa060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.177+0000 7f601156b640 1 -- 172.21.15.120:0/242572 >> 172.21.15.120:0/242572 conn(0x7f600c0b4ba0 msgr2=0x7f600c0b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:28.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.177+0000 7f601156b640 1 -- 172.21.15.120:0/242572 shutdown_connections 2024-02-06T22:18:28.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.177+0000 7f601156b640 1 -- 172.21.15.120:0/242572 wait complete. 2024-02-06T22:18:28.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.178+0000 7f601156b640 10 monclient: build_initial_monmap 2024-02-06T22:18:28.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.178+0000 7f601156b640 10 monclient: monmap: 2024-02-06T22:18:28.185 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:28.185 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:28.185 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:28.185 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:28.185 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:28.185 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:28.185 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:28.185 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:28.185 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:28.185 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:28.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.178+0000 7f601156b640 1 librados: starting msgr at 2024-02-06T22:18:28.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.178+0000 7f601156b640 1 librados: starting objecter 2024-02-06T22:18:28.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.178+0000 7f601156b640 1 Processor -- start 2024-02-06T22:18:28.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 1 -- start start 2024-02-06T22:18:28.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 1 librados: setting wanted keys 2024-02-06T22:18:28.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 1 librados: calling monclient init 2024-02-06T22:18:28.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 10 monclient: init 2024-02-06T22:18:28.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:28.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 10 monclient: _add_conns ranks=[2,1,0] 2024-02-06T22:18:28.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f600c0a9c80 0x7f600c138110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:28.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 10 monclient(hunting): picked mon.noname-c con 0x7f600c0a9c80 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:28.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f600c13f810 0x7f600c1db380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:28.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 10 monclient(hunting): picked mon.noname-b con 0x7f600c13f810 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:28.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f600c142bb0 0x7f600c13b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:28.186 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 10 monclient(hunting): picked mon.noname-a con 0x7f600c142bb0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:28.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:28.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1dae60 con 0x7f600c142bb0 2024-02-06T22:18:28.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:28.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1db8c0 con 0x7f600c13f810 2024-02-06T22:18:28.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:28.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1daa80 con 0x7f600c0a9c80 2024-02-06T22:18:28.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:28.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.179+0000 7f601156b640 10 monclient(hunting): authenticate will time out at 3538.517099s 2024-02-06T22:18:28.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600affd640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f600c0a9c80 0x7f600c138110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:28.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f600c142bb0 0x7f600c13b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:28.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600a7fc640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f600c13f810 0x7f600c1db380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:28.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f600c142bb0 0x7f600c13b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:53360/0 (socket says 172.21.15.120:53360) 2024-02-06T22:18:28.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600b7fe640 1 -- 172.21.15.120:0/344850572 learned_addr learned my addr 172.21.15.120:0/344850572 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:28.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600b7fe640 10 monclient(hunting): get_auth_request con 0x7f600c142bb0 auth_method 0 2024-02-06T22:18:28.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:28.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600b7fe640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:28.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:28.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600a7fc640 10 monclient(hunting): get_auth_request con 0x7f600c13f810 auth_method 0 2024-02-06T22:18:28.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:28.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600a7fc640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:28.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:28.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600affd640 10 monclient(hunting): get_auth_request con 0x7f600c0a9c80 auth_method 0 2024-02-06T22:18:28.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:28.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600affd640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:28.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600affd640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:28.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:28.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:28.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.180+0000 7f600b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:28.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.181+0000 7f600a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:28.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.181+0000 7f600a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:28.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.181+0000 7f600a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:28.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.181+0000 7f600affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:28.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.181+0000 7f600affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:28.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.181+0000 7f600affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:28.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.183+0000 7f600b7fe640 10 monclient(hunting): handle_auth_done global_id 16605 payload 824 2024-02-06T22:18:28.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.183+0000 7f600b7fe640 1 -- 172.21.15.120:0/344850572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f600c0a9c80 msgr2=0x7f600c138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:28.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.183+0000 7f600b7fe640 1 --2- 172.21.15.120:0/344850572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f600c0a9c80 0x7f600c138110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.183+0000 7f600b7fe640 1 -- 172.21.15.120:0/344850572 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f600c13f810 msgr2=0x7f600c1db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:28.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.183+0000 7f600b7fe640 1 --2- 172.21.15.120:0/344850572 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f600c13f810 0x7f600c1db380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.183+0000 7f600b7fe640 10 monclient: _finish_hunting 0 2024-02-06T22:18:28.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.183+0000 7f600b7fe640 1 monclient: found mon.noname-a 2024-02-06T22:18:28.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.183+0000 7f600b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:28.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.183+0000 7f600b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:18:28.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.183+0000 7f600b7fe640 1 -- 172.21.15.120:0/344850572 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6000009400 con 0x7f600c142bb0 2024-02-06T22:18:28.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.185+0000 7f600b7fe640 1 --2- 172.21.15.120:0/344850572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f600c142bb0 0x7f600c13b080 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7f600400cb40 tx=0x7f600400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:28.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.185+0000 7f5ff37fe640 10 client.?.objecter ms_handle_connect 0x7f600c142bb0 2024-02-06T22:18:28.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.185+0000 7f5ff37fe640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:28.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f5ff37fe640 1 -- 172.21.15.120:0/344850572 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600400aca0 con 0x7f600c142bb0 2024-02-06T22:18:28.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f5ff37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:28.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f5ff37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:18:28.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f5ff37fe640 10 monclient: dump: 2024-02-06T22:18:28.192 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:28.192 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:28.192 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.192 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.193 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:28.193 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:28.193 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:28.193 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:28.193 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:28.193 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:28.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f5ff37fe640 10 monclient: _finish_auth 0 2024-02-06T22:18:28.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f5ff37fe640 10 monclient: _check_auth_tickets 2024-02-06T22:18:28.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f5ff37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:28.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f5ff37fe640 1 -- 172.21.15.120:0/344850572 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600400ae00 con 0x7f600c142bb0 2024-02-06T22:18:28.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f5ff37fe640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:28.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f5ff37fe640 1 -- 172.21.15.120:0/344850572 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6004012620 con 0x7f600c142bb0 2024-02-06T22:18:28.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f5ff37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:28.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f5ff37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:18:28.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f5ff37fe640 10 monclient: dump: 2024-02-06T22:18:28.194 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:28.194 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:28.194 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.194 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.194 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:28.194 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:28.194 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:28.194 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:28.194 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:28.194 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:28.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f601156b640 5 monclient: authenticate success, global_id 16605 2024-02-06T22:18:28.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f601156b640 10 monclient: _renew_subs 2024-02-06T22:18:28.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f601156b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:28.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f601156b640 1 -- 172.21.15.120:0/344850572 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600c13a2d0 con 0x7f600c142bb0 2024-02-06T22:18:28.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f60097fa640 4 set_mon_vals no callback set 2024-02-06T22:18:28.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f601156b640 10 client.16605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:28.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f601156b640 10 monclient: _renew_subs 2024-02-06T22:18:28.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f601156b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:28.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f601156b640 1 -- 172.21.15.120:0/344850572 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600c13b7a0 con 0x7f600c142bb0 2024-02-06T22:18:28.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.189+0000 7f601156b640 1 librados: init done 2024-02-06T22:18:28.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.190+0000 7f601156b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:28.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.190+0000 7f601156b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:28.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.190+0000 7f601156b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:28.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.190+0000 7f601156b640 1 -- 172.21.15.120:0/344850572 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f600c04ef90 con 0x7f600c142bb0 2024-02-06T22:18:28.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.208+0000 7f5ff37fe640 1 -- 172.21.15.120:0/344850572 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6004007070 con 0x7f600c142bb0 2024-02-06T22:18:28.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.208+0000 7f5ff37fe640 1 --2- 172.21.15.120:0/344850572 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f5ff806d1f0 0x7f5ff806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:28.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.208+0000 7f600affd640 1 --2- 172.21.15.120:0/344850572 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f5ff806d1f0 0x7f5ff806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:28.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.208+0000 7f600affd640 10 monclient: get_auth_request con 0x7f5ff806d1f0 auth_method 0 2024-02-06T22:18:28.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.209+0000 7f600affd640 1 --2- 172.21.15.120:0/344850572 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f5ff806d1f0 0x7f5ff806f6b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009b40 tx=0x7f5ffc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:28.210 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.209+0000 7f5ff37fe640 10 client.16605.objecter ms_handle_connect 0x7f5ff806d1f0 2024-02-06T22:18:28.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.210+0000 7f5ff37fe640 1 -- 172.21.15.120:0/344850572 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f6004058ca0 con 0x7f600c142bb0 2024-02-06T22:18:28.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.210+0000 7f5ff37fe640 10 client.16605.objecter ms_dispatch 0x7f600c000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:28.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.210+0000 7f5ff37fe640 3 client.16605.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:28.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.210+0000 7f5ff37fe640 3 client.16605.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:28.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.211+0000 7f5ff37fe640 20 client.16605.objecter dump_active .. 0 homeless 2024-02-06T22:18:28.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.441+0000 7f5ff37fe640 1 -- 172.21.15.120:0/344850572 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6004054f40 con 0x7f600c142bb0 2024-02-06T22:18:28.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.441+0000 7f5ff37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:28.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.441+0000 7f5ff37fe640 10 monclient: _finish_command 1 = system:0 2024-02-06T22:18:28.621 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.620+0000 7f601156b640 1 -- 172.21.15.120:0/344850572 --> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f600c14ce80 con 0x7f5ff806d1f0 2024-02-06T22:18:28.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.625+0000 7f5ff37fe640 1 -- 172.21.15.120:0/344850572 <== mgr.4100 v2:172.21.15.135:6800/315664144 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151624 (secure 0 0 0) 0x7f5ffc011a60 con 0x7f5ff806d1f0 2024-02-06T22:18:28.628 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-02-06T22:18:28.629 INFO:teuthology.orchestra.run.smithi120.stderr:dumped all 2024-02-06T22:18:28.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.627+0000 7f601156b640 10 librados: watch_flush enter 2024-02-06T22:18:28.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.627+0000 7f601156b640 10 librados: watch_flush exit 2024-02-06T22:18:28.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.627+0000 7f601156b640 20 client.16605.objecter shutdown clearing up homeless session... 2024-02-06T22:18:28.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.627+0000 7f601156b640 10 client.16605.objecter successfully canceled tick 2024-02-06T22:18:28.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.628+0000 7f601156b640 1 -- 172.21.15.120:0/344850572 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f5ff806d1f0 msgr2=0x7f5ff806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:28.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.628+0000 7f601156b640 1 --2- 172.21.15.120:0/344850572 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f5ff806d1f0 0x7f5ff806f6b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009b40 tx=0x7f5ffc009190 comp rx=0 tx=0).stop 2024-02-06T22:18:28.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.628+0000 7f601156b640 10 monclient: shutdown 2024-02-06T22:18:28.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.628+0000 7f601156b640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:28.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.628+0000 7f601156b640 1 -- 172.21.15.120:0/344850572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f600c142bb0 msgr2=0x7f600c13b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:28.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.628+0000 7f601156b640 1 --2- 172.21.15.120:0/344850572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f600c142bb0 0x7f600c13b080 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7f600400cb40 tx=0x7f600400a060 comp rx=0 tx=0).stop 2024-02-06T22:18:28.632 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.628+0000 7f601156b640 1 -- 172.21.15.120:0/344850572 shutdown_connections 2024-02-06T22:18:28.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.628+0000 7f601156b640 1 --2- 172.21.15.120:0/344850572 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f5ff806d1f0 0x7f5ff806f6b0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.628+0000 7f601156b640 1 --2- 172.21.15.120:0/344850572 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f600c13f810 0x7f600c1db380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.628+0000 7f601156b640 1 --2- 172.21.15.120:0/344850572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f600c142bb0 0x7f600c13b080 unknown :-1 s=CLOSED pgs=3368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.628+0000 7f601156b640 1 --2- 172.21.15.120:0/344850572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f600c0a9c80 0x7f600c138110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.628+0000 7f601156b640 1 -- 172.21.15.120:0/344850572 >> 172.21.15.120:0/344850572 conn(0x7f600c0b4ba0 msgr2=0x7f600c13c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:28.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.628+0000 7f601156b640 1 -- 172.21.15.120:0/344850572 shutdown_connections 2024-02-06T22:18:28.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.628+0000 7f601156b640 1 -- 172.21.15.120:0/344850572 wait complete. 2024-02-06T22:18:28.633 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.628+0000 7f601156b640 1 librados: shutdown 2024-02-06T22:18:28.640 INFO:teuthology.orchestra.run.smithi120.stdout:{"pg_ready":true,"pg_map":{"version":2464,"stamp":"2024-02-06T22:18:28.296919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":583288,"kb_used_data":40952,"kb_used_omap":12,"kb_used_meta":542259,"kb_avail":754391432,"statfs":{"total":773094113280,"available":772496826368,"internally_reserved":0,"allocated":41934848,"data_stored":19968162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":555273776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002824"},"pg_stats":[{"pgid":"12.1c","version":"0'0","reported_seq":2603,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:07.085884+0000","last_change":"2024-02-06T22:18:07.086588+0000","last_active":"2024-02-06T22:18:07.085884+0000","last_peered":"2024-02-06T22:18:07.085884+0000","last_clean":"2024-02-06T22:18:07.085884+0000","last_became_active":"2024-02-06T21:42:50.078423+0000","last_became_peered":"2024-02-06T21:42:50.078423+0000","last_unstale":"2024-02-06T22:18:07.085884+0000","last_undegraded":"2024-02-06T22:18:07.085884+0000","last_fullsized":"2024-02-06T22:18:07.085884+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:07.084448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:07.084448+0000","last_clean_scrub_stamp":"2024-02-06T22:18:07.084448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:20:02.007546+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":2596,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:08.851693+0000","last_change":"2024-02-06T22:18:08.852431+0000","last_active":"2024-02-06T22:18:08.851693+0000","last_peered":"2024-02-06T22:18:08.851693+0000","last_clean":"2024-02-06T22:18:08.851693+0000","last_became_active":"2024-02-06T21:43:19.773686+0000","last_became_peered":"2024-02-06T21:43:19.773686+0000","last_unstale":"2024-02-06T22:18:08.851693+0000","last_undegraded":"2024-02-06T22:18:08.851693+0000","last_fullsized":"2024-02-06T22:18:08.851693+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:08.850206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:08.850206+0000","last_clean_scrub_stamp":"2024-02-06T22:18:08.850206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:08:28.781762+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":2602,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:03.440394+0000","last_change":"2024-02-06T22:18:03.441138+0000","last_active":"2024-02-06T22:18:03.440394+0000","last_peered":"2024-02-06T22:18:03.440394+0000","last_clean":"2024-02-06T22:18:03.440394+0000","last_became_active":"2024-02-06T21:42:53.291194+0000","last_became_peered":"2024-02-06T21:42:53.291194+0000","last_unstale":"2024-02-06T22:18:03.440394+0000","last_undegraded":"2024-02-06T22:18:03.440394+0000","last_fullsized":"2024-02-06T22:18:03.440394+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:03.438950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:03.438950+0000","last_clean_scrub_stamp":"2024-02-06T22:18:03.438950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T07:42:58.828481+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":2592,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:09.618777+0000","last_change":"2024-02-06T22:18:09.619529+0000","last_active":"2024-02-06T22:18:09.618777+0000","last_peered":"2024-02-06T22:18:09.618777+0000","last_clean":"2024-02-06T22:18:09.618777+0000","last_became_active":"2024-02-06T21:42:45.065933+0000","last_became_peered":"2024-02-06T21:42:45.065933+0000","last_unstale":"2024-02-06T22:18:09.618777+0000","last_undegraded":"2024-02-06T22:18:09.618777+0000","last_fullsized":"2024-02-06T22:18:09.618777+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:09.617352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:09.617352+0000","last_clean_scrub_stamp":"2024-02-06T22:18:09.617352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T03:57:49.603460+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":2588,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:05.066332+0000","last_change":"2024-02-06T22:18:05.067575+0000","last_active":"2024-02-06T22:18:05.066332+0000","last_peered":"2024-02-06T22:18:05.066332+0000","last_clean":"2024-02-06T22:18:05.066332+0000","last_became_active":"2024-02-06T21:42:58.548361+0000","last_became_peered":"2024-02-06T21:42:58.548361+0000","last_unstale":"2024-02-06T22:18:05.066332+0000","last_undegraded":"2024-02-06T22:18:05.066332+0000","last_fullsized":"2024-02-06T22:18:05.066332+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:05.064036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:05.064036+0000","last_clean_scrub_stamp":"2024-02-06T22:18:05.064036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T06:28:19.697261+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"1346'8","reported_seq":2608,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:04.930223+0000","last_change":"2024-02-06T22:18:04.931200+0000","last_active":"2024-02-06T22:18:04.930223+0000","last_peered":"2024-02-06T22:18:04.930223+0000","last_clean":"2024-02-06T22:18:04.930223+0000","last_became_active":"2024-02-06T21:43:15.497081+0000","last_became_peered":"2024-02-06T21:43:15.497081+0000","last_unstale":"2024-02-06T22:18:04.930223+0000","last_undegraded":"2024-02-06T22:18:04.930223+0000","last_fullsized":"2024-02-06T22:18:04.930223+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"1346'8","last_scrub_stamp":"2024-02-06T22:18:04.928748+0000","last_deep_scrub":"1346'8","last_deep_scrub_stamp":"2024-02-06T22:18:04.928748+0000","last_clean_scrub_stamp":"2024-02-06T22:18:04.928748+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:29:46.765782+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":2604,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:06.355061+0000","last_change":"2024-02-06T22:18:06.355061+0000","last_active":"2024-02-06T22:18:06.355061+0000","last_peered":"2024-02-06T22:18:06.355061+0000","last_clean":"2024-02-06T22:18:06.355061+0000","last_became_active":"2024-02-06T21:43:13.108638+0000","last_became_peered":"2024-02-06T21:43:13.108638+0000","last_unstale":"2024-02-06T22:18:06.355061+0000","last_undegraded":"2024-02-06T22:18:06.355061+0000","last_fullsized":"2024-02-06T22:18:06.355061+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:06.352885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:06.352885+0000","last_clean_scrub_stamp":"2024-02-06T22:18:06.352885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T09:33:15.254082+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":2611,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:07.783485+0000","last_change":"2024-02-06T22:18:07.783485+0000","last_active":"2024-02-06T22:18:07.783485+0000","last_peered":"2024-02-06T22:18:07.783485+0000","last_clean":"2024-02-06T22:18:07.783485+0000","last_became_active":"2024-02-06T21:43:06.270947+0000","last_became_peered":"2024-02-06T21:43:06.270947+0000","last_unstale":"2024-02-06T22:18:07.783485+0000","last_undegraded":"2024-02-06T22:18:07.783485+0000","last_fullsized":"2024-02-06T22:18:07.783485+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:07.781302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:07.781302+0000","last_clean_scrub_stamp":"2024-02-06T22:18:07.781302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:13:11.117635+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":2604,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:04.405583+0000","last_change":"2024-02-06T22:18:04.405583+0000","last_active":"2024-02-06T22:18:04.405583+0000","last_peered":"2024-02-06T22:18:04.405583+0000","last_clean":"2024-02-06T22:18:04.405583+0000","last_became_active":"2024-02-06T21:42:53.535059+0000","last_became_peered":"2024-02-06T21:42:53.535059+0000","last_unstale":"2024-02-06T22:18:04.405583+0000","last_undegraded":"2024-02-06T22:18:04.405583+0000","last_fullsized":"2024-02-06T22:18:04.405583+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:04.402270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:04.402270+0000","last_clean_scrub_stamp":"2024-02-06T22:18:04.402270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:56:15.260983+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":2591,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:02.109944+0000","last_change":"2024-02-06T22:18:02.109944+0000","last_active":"2024-02-06T22:18:02.109944+0000","last_peered":"2024-02-06T22:18:02.109944+0000","last_clean":"2024-02-06T22:18:02.109944+0000","last_became_active":"2024-02-06T21:42:46.168640+0000","last_became_peered":"2024-02-06T21:42:46.168640+0000","last_unstale":"2024-02-06T22:18:02.109944+0000","last_undegraded":"2024-02-06T22:18:02.109944+0000","last_fullsized":"2024-02-06T22:18:02.109944+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:02.105322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:02.105322+0000","last_clean_scrub_stamp":"2024-02-06T22:18:02.105322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:15:49.183857+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":2587,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:09.847485+0000","last_change":"2024-02-06T22:18:09.848260+0000","last_active":"2024-02-06T22:18:09.847485+0000","last_peered":"2024-02-06T22:18:09.847485+0000","last_clean":"2024-02-06T22:18:09.847485+0000","last_became_active":"2024-02-06T21:43:27.883526+0000","last_became_peered":"2024-02-06T21:43:27.883526+0000","last_unstale":"2024-02-06T22:18:09.847485+0000","last_undegraded":"2024-02-06T22:18:09.847485+0000","last_fullsized":"2024-02-06T22:18:09.847485+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:09.846084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:09.846084+0000","last_clean_scrub_stamp":"2024-02-06T22:18:09.846084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:04:18.520863+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":2592,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:11.392819+0000","last_change":"2024-02-06T22:18:11.393539+0000","last_active":"2024-02-06T22:18:11.392819+0000","last_peered":"2024-02-06T22:18:11.392819+0000","last_clean":"2024-02-06T22:18:11.392819+0000","last_became_active":"2024-02-06T21:42:45.868987+0000","last_became_peered":"2024-02-06T21:42:45.868987+0000","last_unstale":"2024-02-06T22:18:11.392819+0000","last_undegraded":"2024-02-06T22:18:11.392819+0000","last_fullsized":"2024-02-06T22:18:11.392819+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:11.391402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:11.391402+0000","last_clean_scrub_stamp":"2024-02-06T22:18:11.391402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:09:31.623541+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":2589,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:10.572820+0000","last_change":"2024-02-06T22:18:10.573573+0000","last_active":"2024-02-06T22:18:10.572820+0000","last_peered":"2024-02-06T22:18:10.572820+0000","last_clean":"2024-02-06T22:18:10.572820+0000","last_became_active":"2024-02-06T21:42:59.949344+0000","last_became_peered":"2024-02-06T21:42:59.949344+0000","last_unstale":"2024-02-06T22:18:10.572820+0000","last_undegraded":"2024-02-06T22:18:10.572820+0000","last_fullsized":"2024-02-06T22:18:10.572820+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:10.571469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:10.571469+0000","last_clean_scrub_stamp":"2024-02-06T22:18:10.571469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:45:06.783654+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"1346'78","reported_seq":2715,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:03.290419+0000","last_change":"2024-02-06T22:18:03.296110+0000","last_active":"2024-02-06T22:18:03.290419+0000","last_peered":"2024-02-06T22:18:03.290419+0000","last_clean":"2024-02-06T22:18:03.290419+0000","last_became_active":"2024-02-06T21:41:31.305766+0000","last_became_peered":"2024-02-06T21:41:31.305766+0000","last_unstale":"2024-02-06T22:18:03.290419+0000","last_undegraded":"2024-02-06T22:18:03.290419+0000","last_fullsized":"2024-02-06T22:18:03.290419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"1346'78","last_scrub_stamp":"2024-02-06T22:18:03.279798+0000","last_deep_scrub":"1346'78","last_deep_scrub_stamp":"2024-02-06T22:18:03.279798+0000","last_clean_scrub_stamp":"2024-02-06T22:18:03.279798+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:53:47.078250+0000","scrub_duration":415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2603,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:08.602620+0000","last_change":"2024-02-06T22:18:08.603340+0000","last_active":"2024-02-06T22:18:08.602620+0000","last_peered":"2024-02-06T22:18:08.602620+0000","last_clean":"2024-02-06T22:18:08.602620+0000","last_became_active":"2024-02-06T21:41:36.162988+0000","last_became_peered":"2024-02-06T21:41:36.162988+0000","last_unstale":"2024-02-06T22:18:08.602620+0000","last_undegraded":"2024-02-06T22:18:08.602620+0000","last_fullsized":"2024-02-06T22:18:08.602620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:08.601013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:08.601013+0000","last_clean_scrub_stamp":"2024-02-06T22:18:08.601013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T08:34:24.040083+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":2591,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:14.404452+0000","last_change":"2024-02-06T22:18:14.405226+0000","last_active":"2024-02-06T22:18:14.404452+0000","last_peered":"2024-02-06T22:18:14.404452+0000","last_clean":"2024-02-06T22:18:14.404452+0000","last_became_active":"2024-02-06T21:42:56.023045+0000","last_became_peered":"2024-02-06T21:42:56.023045+0000","last_unstale":"2024-02-06T22:18:14.404452+0000","last_undegraded":"2024-02-06T22:18:14.404452+0000","last_fullsized":"2024-02-06T22:18:14.404452+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:14.403089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:14.403089+0000","last_clean_scrub_stamp":"2024-02-06T22:18:14.403089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:46:08.048770+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2605,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:04.647321+0000","last_change":"2024-02-06T22:18:04.648092+0000","last_active":"2024-02-06T22:18:04.647321+0000","last_peered":"2024-02-06T22:18:04.647321+0000","last_clean":"2024-02-06T22:18:04.647321+0000","last_became_active":"2024-02-06T21:41:35.642164+0000","last_became_peered":"2024-02-06T21:41:35.642164+0000","last_unstale":"2024-02-06T22:18:04.647321+0000","last_undegraded":"2024-02-06T22:18:04.647321+0000","last_fullsized":"2024-02-06T22:18:04.647321+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:04.645804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:04.645804+0000","last_clean_scrub_stamp":"2024-02-06T22:18:04.645804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:14:33.592438+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":2586,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:11.620125+0000","last_change":"2024-02-06T22:18:11.620839+0000","last_active":"2024-02-06T22:18:11.620125+0000","last_peered":"2024-02-06T22:18:11.620125+0000","last_clean":"2024-02-06T22:18:11.620125+0000","last_became_active":"2024-02-06T21:43:05.414230+0000","last_became_peered":"2024-02-06T21:43:05.414230+0000","last_unstale":"2024-02-06T22:18:11.620125+0000","last_undegraded":"2024-02-06T22:18:11.620125+0000","last_fullsized":"2024-02-06T22:18:11.620125+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:11.618727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:11.618727+0000","last_clean_scrub_stamp":"2024-02-06T22:18:11.618727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:23:39.124017+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2603,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:08.273819+0000","last_change":"2024-02-06T22:18:08.274562+0000","last_active":"2024-02-06T22:18:08.273819+0000","last_peered":"2024-02-06T22:18:08.273819+0000","last_clean":"2024-02-06T22:18:08.273819+0000","last_became_active":"2024-02-06T21:41:35.603135+0000","last_became_peered":"2024-02-06T21:41:35.603135+0000","last_unstale":"2024-02-06T22:18:08.273819+0000","last_undegraded":"2024-02-06T22:18:08.273819+0000","last_fullsized":"2024-02-06T22:18:08.273819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:08.272390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:08.272390+0000","last_clean_scrub_stamp":"2024-02-06T22:18:08.272390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T03:26:38.104971+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":2592,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:12.361651+0000","last_change":"2024-02-06T22:18:12.362358+0000","last_active":"2024-02-06T22:18:12.361651+0000","last_peered":"2024-02-06T22:18:12.361651+0000","last_clean":"2024-02-06T22:18:12.361651+0000","last_became_active":"2024-02-06T21:42:49.409862+0000","last_became_peered":"2024-02-06T21:42:49.409862+0000","last_unstale":"2024-02-06T22:18:12.361651+0000","last_undegraded":"2024-02-06T22:18:12.361651+0000","last_fullsized":"2024-02-06T22:18:12.361651+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:12.360242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:12.360242+0000","last_clean_scrub_stamp":"2024-02-06T22:18:12.360242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T04:07:41.574898+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2604,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:06.644619+0000","last_change":"2024-02-06T22:18:06.645368+0000","last_active":"2024-02-06T22:18:06.644619+0000","last_peered":"2024-02-06T22:18:06.644619+0000","last_clean":"2024-02-06T22:18:06.644619+0000","last_became_active":"2024-02-06T21:41:35.877657+0000","last_became_peered":"2024-02-06T21:41:35.877657+0000","last_unstale":"2024-02-06T22:18:06.644619+0000","last_undegraded":"2024-02-06T22:18:06.644619+0000","last_fullsized":"2024-02-06T22:18:06.644619+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:06.643162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:06.643162+0000","last_clean_scrub_stamp":"2024-02-06T22:18:06.643162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T06:00:43.093608+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":2610,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:06.757341+0000","last_change":"2024-02-06T22:18:06.758067+0000","last_active":"2024-02-06T22:18:06.757341+0000","last_peered":"2024-02-06T22:18:06.757341+0000","last_clean":"2024-02-06T22:18:06.757341+0000","last_became_active":"2024-02-06T21:42:56.473273+0000","last_became_peered":"2024-02-06T21:42:56.473273+0000","last_unstale":"2024-02-06T22:18:06.757341+0000","last_undegraded":"2024-02-06T22:18:06.757341+0000","last_fullsized":"2024-02-06T22:18:06.757341+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:06.755871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:06.755871+0000","last_clean_scrub_stamp":"2024-02-06T22:18:06.755871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T02:15:33.013548+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2615,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:06.115238+0000","last_change":"2024-02-06T22:18:06.116009+0000","last_active":"2024-02-06T22:18:06.115238+0000","last_peered":"2024-02-06T22:18:06.115238+0000","last_clean":"2024-02-06T22:18:06.115238+0000","last_became_active":"2024-02-06T21:41:35.891748+0000","last_became_peered":"2024-02-06T21:41:35.891748+0000","last_unstale":"2024-02-06T22:18:06.115238+0000","last_undegraded":"2024-02-06T22:18:06.115238+0000","last_fullsized":"2024-02-06T22:18:06.115238+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:06.113709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:06.113709+0000","last_clean_scrub_stamp":"2024-02-06T22:18:06.113709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T07:04:26.757396+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":2592,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:13.427134+0000","last_change":"2024-02-06T22:18:13.427849+0000","last_active":"2024-02-06T22:18:13.427134+0000","last_peered":"2024-02-06T22:18:13.427134+0000","last_clean":"2024-02-06T22:18:13.427134+0000","last_became_active":"2024-02-06T21:42:49.671861+0000","last_became_peered":"2024-02-06T21:42:49.671861+0000","last_unstale":"2024-02-06T22:18:13.427134+0000","last_undegraded":"2024-02-06T22:18:13.427134+0000","last_fullsized":"2024-02-06T22:18:13.427134+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:13.425712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:13.425712+0000","last_clean_scrub_stamp":"2024-02-06T22:18:13.425712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T22:58:31.798313+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2603,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:05.696167+0000","last_change":"2024-02-06T22:18:05.697275+0000","last_active":"2024-02-06T22:18:05.696167+0000","last_peered":"2024-02-06T22:18:05.696167+0000","last_clean":"2024-02-06T22:18:05.696167+0000","last_became_active":"2024-02-06T21:41:35.761324+0000","last_became_peered":"2024-02-06T21:41:35.761324+0000","last_unstale":"2024-02-06T22:18:05.696167+0000","last_undegraded":"2024-02-06T22:18:05.696167+0000","last_fullsized":"2024-02-06T22:18:05.696167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:05.694683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:05.694683+0000","last_clean_scrub_stamp":"2024-02-06T22:18:05.694683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T09:44:19.127541+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":2593,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:09.251212+0000","last_change":"2024-02-06T22:18:09.251930+0000","last_active":"2024-02-06T22:18:09.251212+0000","last_peered":"2024-02-06T22:18:09.251212+0000","last_clean":"2024-02-06T22:18:09.251212+0000","last_became_active":"2024-02-06T21:42:44.691217+0000","last_became_peered":"2024-02-06T21:42:44.691217+0000","last_unstale":"2024-02-06T22:18:09.251212+0000","last_undegraded":"2024-02-06T22:18:09.251212+0000","last_fullsized":"2024-02-06T22:18:09.251212+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:09.249830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:09.249830+0000","last_clean_scrub_stamp":"2024-02-06T22:18:09.249830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T02:30:23.193399+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2627,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:05.780106+0000","last_change":"2024-02-06T22:18:05.780106+0000","last_active":"2024-02-06T22:18:05.780106+0000","last_peered":"2024-02-06T22:18:05.780106+0000","last_clean":"2024-02-06T22:18:05.780106+0000","last_became_active":"2024-02-06T21:41:36.160648+0000","last_became_peered":"2024-02-06T21:41:36.160648+0000","last_unstale":"2024-02-06T22:18:05.780106+0000","last_undegraded":"2024-02-06T22:18:05.780106+0000","last_fullsized":"2024-02-06T22:18:05.780106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:05.778011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:05.778011+0000","last_clean_scrub_stamp":"2024-02-06T22:18:05.778011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T10:03:48.920618+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":2597,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:07.831501+0000","last_change":"2024-02-06T22:18:07.832263+0000","last_active":"2024-02-06T22:18:07.831501+0000","last_peered":"2024-02-06T22:18:07.831501+0000","last_clean":"2024-02-06T22:18:07.831501+0000","last_became_active":"2024-02-06T21:42:52.611794+0000","last_became_peered":"2024-02-06T21:42:52.611794+0000","last_unstale":"2024-02-06T22:18:07.831501+0000","last_undegraded":"2024-02-06T22:18:07.831501+0000","last_fullsized":"2024-02-06T22:18:07.831501+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:07.830068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:07.830068+0000","last_clean_scrub_stamp":"2024-02-06T22:18:07.830068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T00:50:59.856988+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2600,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:07.644715+0000","last_change":"2024-02-06T22:18:07.645433+0000","last_active":"2024-02-06T22:18:07.644715+0000","last_peered":"2024-02-06T22:18:07.644715+0000","last_clean":"2024-02-06T22:18:07.644715+0000","last_became_active":"2024-02-06T21:41:36.139312+0000","last_became_peered":"2024-02-06T21:41:36.139312+0000","last_unstale":"2024-02-06T22:18:07.644715+0000","last_undegraded":"2024-02-06T22:18:07.644715+0000","last_fullsized":"2024-02-06T22:18:07.644715+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:07.643340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:07.643340+0000","last_clean_scrub_stamp":"2024-02-06T22:18:07.643340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T03:45:14.603940+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":2589,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:03.249242+0000","last_change":"2024-02-06T22:18:03.249995+0000","last_active":"2024-02-06T22:18:03.249242+0000","last_peered":"2024-02-06T22:18:03.249242+0000","last_clean":"2024-02-06T22:18:03.249242+0000","last_became_active":"2024-02-06T21:42:49.708820+0000","last_became_peered":"2024-02-06T21:42:49.708820+0000","last_unstale":"2024-02-06T22:18:03.249242+0000","last_undegraded":"2024-02-06T22:18:03.249242+0000","last_fullsized":"2024-02-06T22:18:03.249242+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:03.247707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:03.247707+0000","last_clean_scrub_stamp":"2024-02-06T22:18:03.247707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:57:12.568196+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":2601,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:07.411265+0000","last_change":"2024-02-06T22:18:07.411999+0000","last_active":"2024-02-06T22:18:07.411265+0000","last_peered":"2024-02-06T22:18:07.411265+0000","last_clean":"2024-02-06T22:18:07.411265+0000","last_became_active":"2024-02-06T21:43:13.123786+0000","last_became_peered":"2024-02-06T21:43:13.123786+0000","last_unstale":"2024-02-06T22:18:07.411265+0000","last_undegraded":"2024-02-06T22:18:07.411265+0000","last_fullsized":"2024-02-06T22:18:07.411265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:07.409811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:07.409811+0000","last_clean_scrub_stamp":"2024-02-06T22:18:07.409811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-07T23:21:14.231035+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":2587,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:15.463705+0000","last_change":"2024-02-06T22:18:15.464406+0000","last_active":"2024-02-06T22:18:15.463705+0000","last_peered":"2024-02-06T22:18:15.463705+0000","last_clean":"2024-02-06T22:18:15.463705+0000","last_became_active":"2024-02-06T21:42:56.865960+0000","last_became_peered":"2024-02-06T21:42:56.865960+0000","last_unstale":"2024-02-06T22:18:15.463705+0000","last_undegraded":"2024-02-06T22:18:15.463705+0000","last_fullsized":"2024-02-06T22:18:15.463705+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:15.462326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:15.462326+0000","last_clean_scrub_stamp":"2024-02-06T22:18:15.462326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T09:50:58.035480+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":2591,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:10.314784+0000","last_change":"2024-02-06T22:18:10.315485+0000","last_active":"2024-02-06T22:18:10.314784+0000","last_peered":"2024-02-06T22:18:10.314784+0000","last_clean":"2024-02-06T22:18:10.314784+0000","last_became_active":"2024-02-06T21:42:45.706432+0000","last_became_peered":"2024-02-06T21:42:45.706432+0000","last_unstale":"2024-02-06T22:18:10.314784+0000","last_undegraded":"2024-02-06T22:18:10.314784+0000","last_fullsized":"2024-02-06T22:18:10.314784+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:10.313391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:10.313391+0000","last_clean_scrub_stamp":"2024-02-06T22:18:10.313391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T02:52:35.820709+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":2588,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:03.974258+0000","last_change":"2024-02-06T22:18:03.975083+0000","last_active":"2024-02-06T22:18:03.974258+0000","last_peered":"2024-02-06T22:18:03.974258+0000","last_clean":"2024-02-06T22:18:03.974258+0000","last_became_active":"2024-02-06T21:43:08.871254+0000","last_became_peered":"2024-02-06T21:43:08.871254+0000","last_unstale":"2024-02-06T22:18:03.974258+0000","last_undegraded":"2024-02-06T22:18:03.974258+0000","last_fullsized":"2024-02-06T22:18:03.974258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:03.972434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:03.972434+0000","last_clean_scrub_stamp":"2024-02-06T22:18:03.972434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T07:14:44.628388+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":2599,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:05.412669+0000","last_change":"2024-02-06T22:18:05.414020+0000","last_active":"2024-02-06T22:18:05.412669+0000","last_peered":"2024-02-06T22:18:05.412669+0000","last_clean":"2024-02-06T22:18:05.412669+0000","last_became_active":"2024-02-06T21:42:55.034062+0000","last_became_peered":"2024-02-06T21:42:55.034062+0000","last_unstale":"2024-02-06T22:18:05.412669+0000","last_undegraded":"2024-02-06T22:18:05.412669+0000","last_fullsized":"2024-02-06T22:18:05.412669+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:05.408421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:05.408421+0000","last_clean_scrub_stamp":"2024-02-06T22:18:05.408421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T07:02:22.296493+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":2591,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:05.913920+0000","last_change":"2024-02-06T22:18:05.915147+0000","last_active":"2024-02-06T22:18:05.913920+0000","last_peered":"2024-02-06T22:18:05.913920+0000","last_clean":"2024-02-06T22:18:05.913920+0000","last_became_active":"2024-02-06T21:43:15.861140+0000","last_became_peered":"2024-02-06T21:43:15.861140+0000","last_unstale":"2024-02-06T22:18:05.913920+0000","last_undegraded":"2024-02-06T22:18:05.913920+0000","last_fullsized":"2024-02-06T22:18:05.913920+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:05.911716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:05.911716+0000","last_clean_scrub_stamp":"2024-02-06T22:18:05.911716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T10:00:09.737769+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":2590,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:04.077011+0000","last_change":"2024-02-06T22:18:04.077773+0000","last_active":"2024-02-06T22:18:04.077011+0000","last_peered":"2024-02-06T22:18:04.077011+0000","last_clean":"2024-02-06T22:18:04.077011+0000","last_became_active":"2024-02-06T21:42:55.168677+0000","last_became_peered":"2024-02-06T21:42:55.168677+0000","last_unstale":"2024-02-06T22:18:04.077011+0000","last_undegraded":"2024-02-06T22:18:04.077011+0000","last_fullsized":"2024-02-06T22:18:04.077011+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:04.072119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:04.072119+0000","last_clean_scrub_stamp":"2024-02-06T22:18:04.072119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T10:11:31.777405+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":2605,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:08.124705+0000","last_change":"2024-02-06T22:18:08.125409+0000","last_active":"2024-02-06T22:18:08.124705+0000","last_peered":"2024-02-06T22:18:08.124705+0000","last_clean":"2024-02-06T22:18:08.124705+0000","last_became_active":"2024-02-06T21:42:51.993812+0000","last_became_peered":"2024-02-06T21:42:51.993812+0000","last_unstale":"2024-02-06T22:18:08.124705+0000","last_undegraded":"2024-02-06T22:18:08.124705+0000","last_fullsized":"2024-02-06T22:18:08.124705+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:08.122791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:08.122791+0000","last_clean_scrub_stamp":"2024-02-06T22:18:08.122791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:14:49.324230+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":2589,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:08.789159+0000","last_change":"2024-02-06T22:18:08.789946+0000","last_active":"2024-02-06T22:18:08.789159+0000","last_peered":"2024-02-06T22:18:08.789159+0000","last_clean":"2024-02-06T22:18:08.789159+0000","last_became_active":"2024-02-06T21:44:00.992836+0000","last_became_peered":"2024-02-06T21:44:00.992836+0000","last_unstale":"2024-02-06T22:18:08.789159+0000","last_undegraded":"2024-02-06T22:18:08.789159+0000","last_fullsized":"2024-02-06T22:18:08.789159+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:08.784789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:08.784789+0000","last_clean_scrub_stamp":"2024-02-06T22:18:08.784789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T07:36:43.922268+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":2597,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:08.433904+0000","last_change":"2024-02-06T22:18:08.434670+0000","last_active":"2024-02-06T22:18:08.433904+0000","last_peered":"2024-02-06T22:18:08.433904+0000","last_clean":"2024-02-06T22:18:08.433904+0000","last_became_active":"2024-02-06T21:43:14.057258+0000","last_became_peered":"2024-02-06T21:43:14.057258+0000","last_unstale":"2024-02-06T22:18:08.433904+0000","last_undegraded":"2024-02-06T22:18:08.433904+0000","last_fullsized":"2024-02-06T22:18:08.433904+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:08.430553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:08.430553+0000","last_clean_scrub_stamp":"2024-02-06T22:18:08.430553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T05:35:07.114425+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":2597,"reported_epoch":1346,"state":"active+clean","last_fresh":"2024-02-06T22:18:06.802246+0000","last_change":"2024-02-06T22:18:06.802974+0000","last_active":"2024-02-06T22:18:06.802246+0000","last_peered":"2024-02-06T22:18:06.802246+0000","last_clean":"2024-02-06T22:18:06.802246+0000","last_became_active":"2024-02-06T21:42:49.754000+0000","last_became_peered":"2024-02-06T21:42:49.754000+0000","last_unstale":"2024-02-06T22:18:06.802246+0000","last_undegraded":"2024-02-06T22:18:06.802246+0000","last_fullsized":"2024-02-06T22:18:06.802246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-06T22:18:06.800806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-06T22:18:06.800806+0000","last_clean_scrub_stamp":"2024-02-06T22:18:06.800806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-08T01:46:44.159189+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":18,"seq":77309411773,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72688,"kb_used_data":5416,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299152,"statfs":{"total":96636764160,"available":96562331648,"internally_reserved":0,"allocated":5545984,"data_stored":2794404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:41 2024","interfaces":[{"interface":"back","average":{"1min":15.664999999999999,"5min":29.364000000000001,"15min":24.5},"min":{"1min":5.9740000000000002,"5min":4.96,"15min":4.5209999999999999},"max":{"1min":76.638000000000005,"5min":223.274,"15min":549.32299999999998},"last":12.57},{"interface":"front","average":{"1min":20.262,"5min":35.253999999999998,"15min":29.021999999999998},"min":{"1min":10.795999999999999,"5min":7.9969999999999999,"15min":5.4900000000000002},"max":{"1min":78.369,"5min":224.31700000000001,"15min":368.31200000000001},"last":14.205}]},{"osd":2,"last update":"Tue Feb 6 22:18:10 2024","interfaces":[{"interface":"back","average":{"1min":18.036999999999999,"5min":30.946000000000002,"15min":25.704000000000001},"min":{"1min":11.891,"5min":5.9960000000000004,"15min":5.1520000000000001},"max":{"1min":30.402000000000001,"5min":205.28800000000001,"15min":387.75299999999999},"last":13.382999999999999},{"interface":"front","average":{"1min":10.914999999999999,"5min":28.135999999999999,"15min":28.885000000000002},"min":{"1min":5.492,"5min":5.3460000000000001,"15min":5.3460000000000001},"max":{"1min":25.692,"5min":220.31899999999999,"15min":393.44499999999999},"last":8.1300000000000008}]},{"osd":3,"last update":"Tue Feb 6 22:17:55 2024","interfaces":[{"interface":"back","average":{"1min":11.712999999999999,"5min":24.257999999999999,"15min":21.895},"min":{"1min":6.1520000000000001,"5min":5.3899999999999997,"15min":5.3899999999999997},"max":{"1min":68.462999999999994,"5min":359.19299999999998,"15min":359.19299999999998},"last":7.5019999999999998},{"interface":"front","average":{"1min":26.09,"5min":35.533999999999999,"15min":31.292999999999999},"min":{"1min":11.449,"5min":8.5500000000000007,"15min":7.8010000000000002},"max":{"1min":102.22199999999999,"5min":266.40600000000001,"15min":462.86200000000002},"last":14.960000000000001}]},{"osd":4,"last update":"Tue Feb 6 22:17:24 2024","interfaces":[{"interface":"back","average":{"1min":25.393999999999998,"5min":39.593000000000004,"15min":29.221},"min":{"1min":12.670999999999999,"5min":9.0559999999999992,"15min":5.9539999999999997},"max":{"1min":85.082999999999998,"5min":535.22500000000002,"15min":535.22500000000002},"last":15.685},{"interface":"front","average":{"1min":12.597,"5min":28.677,"15min":22.689},"min":{"1min":7.5220000000000002,"5min":6.3010000000000002,"15min":5.4649999999999999},"max":{"1min":49.466000000000001,"5min":473.69799999999998,"15min":473.69799999999998},"last":7.0789999999999997}]},{"osd":5,"last update":"Tue Feb 6 22:17:41 2024","interfaces":[{"interface":"back","average":{"1min":27.221,"5min":36.164999999999999,"15min":32.499000000000002},"min":{"1min":14.074999999999999,"5min":8.0459999999999994,"15min":6.4550000000000001},"max":{"1min":90.605999999999995,"5min":260.74400000000003,"15min":454.49099999999999},"last":16.419},{"interface":"front","average":{"1min":17.254000000000001,"5min":25.736999999999998,"15min":26.802},"min":{"1min":8.1069999999999993,"5min":6.8010000000000002,"15min":6.109},"max":{"1min":70.180000000000007,"5min":147.011,"15min":539.04700000000003},"last":11.089}]},{"osd":6,"last update":"Tue Feb 6 22:17:24 2024","interfaces":[{"interface":"back","average":{"1min":27.564,"5min":40.116999999999997,"15min":37.177},"min":{"1min":13.347,"5min":12.289,"15min":10.353999999999999},"max":{"1min":89.951999999999998,"5min":268.73399999999998,"15min":482.846},"last":17.263999999999999},{"interface":"front","average":{"1min":16.326000000000001,"5min":27.145,"15min":24.978000000000002},"min":{"1min":8.7759999999999998,"5min":6.9349999999999996,"15min":6.9349999999999996},"max":{"1min":51.412999999999997,"5min":186.95500000000001,"15min":322.64999999999998},"last":8.7850000000000001}]},{"osd":7,"last update":"Tue Feb 6 22:17:41 2024","interfaces":[{"interface":"back","average":{"1min":15.381,"5min":27.943000000000001,"15min":30.135000000000002},"min":{"1min":6.1669999999999998,"5min":6.1669999999999998,"15min":6.1669999999999998},"max":{"1min":75.899000000000001,"5min":249.69200000000001,"15min":452.22399999999999},"last":10.367000000000001},{"interface":"front","average":{"1min":31.131,"5min":42.412999999999997,"15min":35.392000000000003},"min":{"1min":13.923999999999999,"5min":10.964,"15min":7.7030000000000003},"max":{"1min":99.406000000000006,"5min":292.94600000000003,"15min":563.26900000000001},"last":18.009}]}]},{"osd":6,"up_from":17,"seq":73014444478,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72296,"kb_used_data":5024,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299544,"statfs":{"total":96636764160,"available":96562733056,"internally_reserved":0,"allocated":5144576,"data_stored":2396564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":14.135,"5min":21.481000000000002,"15min":17.268000000000001},"min":{"1min":4.3339999999999996,"5min":4.3230000000000004,"15min":4.3230000000000004},"max":{"1min":59.670999999999999,"5min":188.761,"15min":218.85400000000001},"last":9.6180000000000003},{"interface":"front","average":{"1min":11.811,"5min":24.353000000000002,"15min":17.314},"min":{"1min":5.1559999999999997,"5min":5.1559999999999997,"15min":4.8449999999999998},"max":{"1min":38.082999999999998,"5min":193.68000000000001,"15min":193.68000000000001},"last":11.132999999999999}]},{"osd":1,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":15.811999999999999,"5min":23.908999999999999,"15min":21.940000000000001},"min":{"1min":6.431,"5min":6.1459999999999999,"15min":5.109},"max":{"1min":54.000999999999998,"5min":180.447,"15min":204.64500000000001},"last":8.3859999999999992},{"interface":"front","average":{"1min":17.199000000000002,"5min":23.283000000000001,"15min":24.012},"min":{"1min":6.7949999999999999,"5min":6.7949999999999999,"15min":6.1070000000000002},"max":{"1min":60.319000000000003,"5min":168.404,"15min":460.71600000000001},"last":18.719000000000001}]},{"osd":2,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":19.161000000000001,"5min":36.042000000000002,"15min":26.562999999999999},"min":{"1min":8.0570000000000004,"5min":6.9359999999999999,"15min":5.5330000000000004},"max":{"1min":72.200000000000003,"5min":794.04200000000003,"15min":794.04200000000003},"last":12.048},{"interface":"front","average":{"1min":19.587,"5min":43.002000000000002,"15min":27.474},"min":{"1min":11.856,"5min":6.2999999999999998,"15min":5.601},"max":{"1min":62.237000000000002,"5min":795.17499999999995,"15min":795.17499999999995},"last":15.659000000000001}]},{"osd":3,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":19.741,"5min":32.057000000000002,"15min":24.117000000000001},"min":{"1min":10.901999999999999,"5min":7.718,"15min":5.9240000000000004},"max":{"1min":72.951999999999998,"5min":317.899,"15min":317.899},"last":13.753},{"interface":"front","average":{"1min":20.227,"5min":29.085999999999999,"15min":21.300999999999998},"min":{"1min":10.553000000000001,"5min":6.468,"15min":6.0380000000000003},"max":{"1min":65.974000000000004,"5min":405.108,"15min":405.108},"last":16.899000000000001}]},{"osd":4,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":10.769,"5min":23.568999999999999,"15min":22.443999999999999},"min":{"1min":5.8940000000000001,"5min":5.7960000000000003,"15min":5.7960000000000003},"max":{"1min":30.241,"5min":231.55699999999999,"15min":243.667},"last":9.1920000000000002},{"interface":"front","average":{"1min":14.613,"5min":30.960000000000001,"15min":27.713000000000001},"min":{"1min":8.641,"5min":7.7160000000000002,"15min":7.3399999999999999},"max":{"1min":39.484999999999999,"5min":284.37,"15min":339.34399999999999},"last":13.026}]},{"osd":5,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":20.939,"5min":24.574999999999999,"15min":22.995000000000001},"min":{"1min":11.087999999999999,"5min":9.2929999999999993,"15min":7.5990000000000002},"max":{"1min":67.230000000000004,"5min":157.095,"15min":240.20500000000001},"last":17.795999999999999},{"interface":"front","average":{"1min":23.254999999999999,"5min":31.707999999999998,"15min":27.795000000000002},"min":{"1min":14.375,"5min":13.481,"15min":9.9260000000000002},"max":{"1min":69.085999999999999,"5min":202.673,"15min":223.42500000000001},"last":20.355}]},{"osd":7,"last update":"Tue Feb 6 22:17:51 2024","interfaces":[{"interface":"back","average":{"1min":23.655999999999999,"5min":30.402999999999999,"15min":23.09},"min":{"1min":15.859,"5min":11.272,"15min":7.4580000000000002},"max":{"1min":70.263999999999996,"5min":190.77799999999999,"15min":224.63},"last":19.545999999999999},{"interface":"front","average":{"1min":21.408999999999999,"5min":31.376999999999999,"15min":27.039999999999999},"min":{"1min":8.7430000000000003,"5min":8.7430000000000003,"15min":8.7430000000000003},"max":{"1min":74.259,"5min":203.26499999999999,"15min":220.893},"last":14.84}]}]},{"osd":5,"up_from":17,"seq":73014444481,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72684,"kb_used_data":5412,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299156,"statfs":{"total":96636764160,"available":96562335744,"internally_reserved":0,"allocated":5541888,"data_stored":2794399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":14.396000000000001,"5min":19.771999999999998,"15min":16.498999999999999},"min":{"1min":6.2709999999999999,"5min":5.6180000000000003,"15min":4.5090000000000003},"max":{"1min":72.215999999999994,"5min":115.977,"15min":350.07600000000002},"last":9.3640000000000008},{"interface":"front","average":{"1min":20.567,"5min":26.533999999999999,"15min":22.433},"min":{"1min":9.1340000000000003,"5min":7.8810000000000002,"15min":6.133},"max":{"1min":81.966999999999999,"5min":128.446,"15min":351.94},"last":10.558999999999999}]},{"osd":1,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":16.960000000000001,"5min":22.443000000000001,"15min":21.648},"min":{"1min":7.9630000000000001,"5min":7.0830000000000002,"15min":6.2359999999999998},"max":{"1min":46.289000000000001,"5min":158.76400000000001,"15min":275.12},"last":9.9740000000000002},{"interface":"front","average":{"1min":19.393000000000001,"5min":24.099,"15min":22.812999999999999},"min":{"1min":8.5939999999999994,"5min":8.1590000000000007,"15min":4.6520000000000001},"max":{"1min":61.463000000000001,"5min":112.78700000000001,"15min":369.73200000000003},"last":17.492000000000001}]},{"osd":2,"last update":"Tue Feb 6 22:18:21 2024","interfaces":[{"interface":"back","average":{"1min":11.407,"5min":43.628999999999998,"15min":27.379000000000001},"min":{"1min":7.1769999999999996,"5min":5.7140000000000004,"15min":5.7140000000000004},"max":{"1min":33.539000000000001,"5min":2330.5610000000001,"15min":2330.5610000000001},"last":7.0960000000000001},{"interface":"front","average":{"1min":16.225999999999999,"5min":51.600000000000001,"15min":31.959},"min":{"1min":8.0999999999999996,"5min":7.3230000000000004,"15min":7.1390000000000002},"max":{"1min":36.869,"5min":2614.4409999999998,"15min":2614.4409999999998},"last":13.798999999999999}]},{"osd":3,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":18.696000000000002,"5min":23.902000000000001,"15min":24.218},"min":{"1min":7.8049999999999997,"5min":6.125,"15min":6.125},"max":{"1min":62.171999999999997,"5min":156.80000000000001,"15min":393.55200000000002},"last":15.066000000000001},{"interface":"front","average":{"1min":14.869,"5min":19.66,"15min":21.309999999999999},"min":{"1min":6.1159999999999997,"5min":5.3819999999999997,"15min":5.3819999999999997},"max":{"1min":63.813000000000002,"5min":104.755,"15min":376.113},"last":16.251999999999999}]},{"osd":4,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":17.641999999999999,"5min":21.885000000000002,"15min":23.100999999999999},"min":{"1min":6.5650000000000004,"5min":6.093,"15min":6.093},"max":{"1min":84.843999999999994,"5min":105.532,"15min":426.77300000000002},"last":8.7710000000000008},{"interface":"front","average":{"1min":18.773,"5min":21.994,"15min":24.087},"min":{"1min":7.4870000000000001,"5min":7.3819999999999997,"15min":5.8330000000000002},"max":{"1min":50.149000000000001,"5min":90.284000000000006,"15min":419.16000000000003},"last":11.461}]},{"osd":6,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":20.283000000000001,"5min":29.465,"15min":28.431999999999999},"min":{"1min":6.9560000000000004,"5min":6.9560000000000004,"15min":6.9560000000000004},"max":{"1min":85.828000000000003,"5min":219.14099999999999,"15min":392.71600000000001},"last":18.175000000000001},{"interface":"front","average":{"1min":21.872,"5min":25.550000000000001,"15min":26.341000000000001},"min":{"1min":9.8740000000000006,"5min":7.2350000000000003,"15min":7.2350000000000003},"max":{"1min":65.367000000000004,"5min":152.69900000000001,"15min":377.31400000000002},"last":14.483000000000001}]},{"osd":7,"last update":"Tue Feb 6 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":19.959,"5min":25.904,"15min":25.827999999999999},"min":{"1min":9.0510000000000002,"5min":9.0510000000000002,"15min":9.0510000000000002},"max":{"1min":63.646000000000001,"5min":124.711,"15min":377.87400000000002},"last":12.163},{"interface":"front","average":{"1min":19.725000000000001,"5min":27.969999999999999,"15min":24.045999999999999},"min":{"1min":7.8940000000000001,"5min":7.8940000000000001,"15min":6.7549999999999999},"max":{"1min":48.325000000000003,"5min":152.59399999999999,"15min":338.803},"last":16.882999999999999}]}]},{"osd":4,"up_from":17,"seq":73014444477,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72292,"kb_used_data":5020,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299548,"statfs":{"total":96636764160,"available":96562737152,"internally_reserved":0,"allocated":5140480,"data_stored":2396559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:24 2024","interfaces":[{"interface":"back","average":{"1min":8.2439999999999998,"5min":17.853000000000002,"15min":16.888999999999999},"min":{"1min":4.907,"5min":4.4169999999999998,"15min":4.3899999999999997},"max":{"1min":19.225999999999999,"5min":167.291,"15min":216.86600000000001},"last":5.3449999999999998},{"interface":"front","average":{"1min":9.9450000000000003,"5min":22.707999999999998,"15min":18.631},"min":{"1min":4.742,"5min":4.6920000000000002,"15min":4.6829999999999998},"max":{"1min":23.745000000000001,"5min":231.82599999999999,"15min":287.97300000000001},"last":8.8670000000000009}]},{"osd":1,"last update":"Tue Feb 6 22:18:07 2024","interfaces":[{"interface":"back","average":{"1min":12.593,"5min":23.126999999999999,"15min":27.245999999999999},"min":{"1min":8.3450000000000006,"5min":6.2949999999999999,"15min":5.1539999999999999},"max":{"1min":20.579000000000001,"5min":207.22399999999999,"15min":678.16999999999996},"last":13.997},{"interface":"front","average":{"1min":13.08,"5min":22.274999999999999,"15min":27.780000000000001},"min":{"1min":6.9400000000000004,"5min":6.5209999999999999,"15min":6.2599999999999998},"max":{"1min":21.433,"5min":203.19999999999999,"15min":667.16300000000001},"last":10.541}]},{"osd":2,"last update":"Tue Feb 6 22:18:07 2024","interfaces":[{"interface":"back","average":{"1min":13.991,"5min":58.281999999999996,"15min":31.885000000000002},"min":{"1min":9.7469999999999999,"5min":6.601,"15min":6.601},"max":{"1min":23.986000000000001,"5min":2277.7020000000002,"15min":2277.7020000000002},"last":12.996},{"interface":"front","average":{"1min":17.437000000000001,"5min":63.363999999999997,"15min":38.143999999999998},"min":{"1min":13.422000000000001,"5min":9.4559999999999995,"15min":9.4559999999999995},"max":{"1min":29.707999999999998,"5min":2279.0419999999999,"15min":2279.0419999999999},"last":16.32}]},{"osd":3,"last update":"Tue Feb 6 22:18:07 2024","interfaces":[{"interface":"back","average":{"1min":20.059000000000001,"5min":28.332000000000001,"15min":26.408999999999999},"min":{"1min":11.769,"5min":6.3559999999999999,"15min":6.3559999999999999},"max":{"1min":35.295999999999999,"5min":214.49000000000001,"15min":338.73399999999998},"last":15.565},{"interface":"front","average":{"1min":12.236000000000001,"5min":29.658000000000001,"15min":22.356999999999999},"min":{"1min":8.1140000000000008,"5min":6.9610000000000003,"15min":5.5910000000000002},"max":{"1min":28.23,"5min":225.64699999999999,"15min":255.14099999999999},"last":9.8339999999999996}]},{"osd":5,"last update":"Tue Feb 6 22:18:07 2024","interfaces":[{"interface":"back","average":{"1min":17.716000000000001,"5min":33.688000000000002,"15min":27.364000000000001},"min":{"1min":11.81,"5min":10.756,"15min":7.2999999999999998},"max":{"1min":32.183999999999997,"5min":248.70500000000001,"15min":263.44799999999998},"last":14.807},{"interface":"front","average":{"1min":12.093,"5min":26.82,"15min":24.356000000000002},"min":{"1min":7.5999999999999996,"5min":6.476,"15min":6.1550000000000002},"max":{"1min":25.489000000000001,"5min":208.02699999999999,"15min":809.58100000000002},"last":11.119999999999999}]},{"osd":6,"last update":"Tue Feb 6 22:18:07 2024","interfaces":[{"interface":"back","average":{"1min":12.262,"5min":21.992000000000001,"15min":22.829999999999998},"min":{"1min":6.9820000000000002,"5min":6.2930000000000001,"15min":6.2930000000000001},"max":{"1min":22.140000000000001,"5min":145.339,"15min":400.46800000000002},"last":7.9569999999999999},{"interface":"front","average":{"1min":21.379000000000001,"5min":31.991,"15min":31.321999999999999},"min":{"1min":14.257999999999999,"5min":11.411,"15min":10.502000000000001},"max":{"1min":58.033999999999999,"5min":269.44200000000001,"15min":269.44200000000001},"last":17.997}]},{"osd":7,"last update":"Tue Feb 6 22:18:07 2024","interfaces":[{"interface":"back","average":{"1min":24.657,"5min":35.323999999999998,"15min":24.632000000000001},"min":{"1min":15.023999999999999,"5min":8.9009999999999998,"15min":6.6239999999999997},"max":{"1min":56.856000000000002,"5min":287.61599999999999,"15min":287.61599999999999},"last":17.212},{"interface":"front","average":{"1min":15.837999999999999,"5min":25.184000000000001,"15min":24.452000000000002},"min":{"1min":8.8460000000000001,"5min":7.0499999999999998,"15min":7.0499999999999998},"max":{"1min":27.827999999999999,"5min":215.26400000000001,"15min":218.876},"last":12.279999999999999}]}]},{"osd":3,"up_from":17,"seq":73014444477,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72292,"kb_used_data":5020,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299548,"statfs":{"total":96636764160,"available":96562737152,"internally_reserved":0,"allocated":5140480,"data_stored":2396559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:35 2024","interfaces":[{"interface":"back","average":{"1min":15.148999999999999,"5min":19.969999999999999,"15min":18.234000000000002},"min":{"1min":5.6100000000000003,"5min":4.5270000000000001,"15min":4.173},"max":{"1min":37.642000000000003,"5min":178.14400000000001,"15min":233.84200000000001},"last":12.132999999999999},{"interface":"front","average":{"1min":17.673999999999999,"5min":22.867000000000001,"15min":22.109999999999999},"min":{"1min":7.1299999999999999,"5min":5.3949999999999996,"15min":5.3949999999999996},"max":{"1min":45.851999999999997,"5min":224.822,"15min":224.822},"last":8.9320000000000004}]},{"osd":1,"last update":"Tue Feb 6 22:17:28 2024","interfaces":[{"interface":"back","average":{"1min":13.183999999999999,"5min":19.210000000000001,"15min":21.574000000000002},"min":{"1min":6.7039999999999997,"5min":6.7039999999999997,"15min":6.7039999999999997},"max":{"1min":21.882999999999999,"5min":169.31100000000001,"15min":273.12700000000001},"last":7.859},{"interface":"front","average":{"1min":16.084,"5min":25.670000000000002,"15min":25.431999999999999},"min":{"1min":10.845000000000001,"5min":10.845000000000001,"15min":9.2880000000000003},"max":{"1min":24.265000000000001,"5min":205.495,"15min":322.70499999999998},"last":9.5299999999999994}]},{"osd":2,"last update":"Tue Feb 6 22:17:35 2024","interfaces":[{"interface":"back","average":{"1min":25.116,"5min":28.814,"15min":26.242999999999999},"min":{"1min":11.129,"5min":11.129,"15min":9.1999999999999993},"max":{"1min":59.799999999999997,"5min":240.316,"15min":282.40699999999998},"last":10.282},{"interface":"front","average":{"1min":11.965,"5min":28.228999999999999,"15min":21.456},"min":{"1min":5.8979999999999997,"5min":5.8419999999999996,"15min":5.1550000000000002},"max":{"1min":39.423000000000002,"5min":998.23500000000001,"15min":998.23500000000001},"last":14.055}]},{"osd":4,"last update":"Tue Feb 6 22:18:22 2024","interfaces":[{"interface":"back","average":{"1min":15.164,"5min":18.533000000000001,"15min":19.475999999999999},"min":{"1min":7.0519999999999996,"5min":6.1109999999999998,"15min":5.8339999999999996},"max":{"1min":53.545000000000002,"5min":162.65299999999999,"15min":189.12},"last":7.0640000000000001},{"interface":"front","average":{"1min":29.140000000000001,"5min":29.943999999999999,"15min":30.077000000000002},"min":{"1min":13.644,"5min":11.710000000000001,"15min":11.045},"max":{"1min":120.91500000000001,"5min":176.35400000000001,"15min":235.315},"last":15.541}]},{"osd":5,"last update":"Tue Feb 6 22:17:35 2024","interfaces":[{"interface":"back","average":{"1min":29.649000000000001,"5min":32.82,"15min":29.718},"min":{"1min":10.526,"5min":10.087999999999999,"15min":8.9920000000000009},"max":{"1min":77.253,"5min":264.06999999999999,"15min":626.27700000000004},"last":11.483000000000001},{"interface":"front","average":{"1min":37.348999999999997,"5min":37.712000000000003,"15min":32.811999999999998},"min":{"1min":13.699,"5min":13.699,"15min":13.699},"max":{"1min":112.84699999999999,"5min":284.88299999999998,"15min":614.85400000000004},"last":17.945}]},{"osd":6,"last update":"Tue Feb 6 22:17:54 2024","interfaces":[{"interface":"back","average":{"1min":33.225999999999999,"5min":29.445,"15min":22.18},"min":{"1min":16.498000000000001,"5min":6.6230000000000002,"15min":6.218},"max":{"1min":75.498000000000005,"5min":177.40700000000001,"15min":234.43700000000001},"last":16.341999999999999},{"interface":"front","average":{"1min":19.661999999999999,"5min":30.109000000000002,"15min":28.652000000000001},"min":{"1min":7.258,"5min":7.258,"15min":7.258},"max":{"1min":66.543999999999997,"5min":209.25800000000001,"15min":232.30000000000001},"last":12.859999999999999}]},{"osd":7,"last update":"Tue Feb 6 22:17:54 2024","interfaces":[{"interface":"back","average":{"1min":20.655000000000001,"5min":23.515000000000001,"15min":22.335999999999999},"min":{"1min":9.3759999999999994,"5min":7.3579999999999997,"15min":7.3579999999999997},"max":{"1min":75.813999999999993,"5min":173.642,"15min":235.012},"last":13.385999999999999},{"interface":"front","average":{"1min":36.191000000000003,"5min":34.972999999999999,"15min":31.428999999999998},"min":{"1min":17.329000000000001,"5min":14.802,"15min":12.015000000000001},"max":{"1min":117.54600000000001,"5min":206.87,"15min":233.11199999999999},"last":17.106000000000002}]}]},{"osd":2,"up_from":17,"seq":73014444477,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72292,"kb_used_data":5020,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299548,"statfs":{"total":96636764160,"available":96562737152,"internally_reserved":0,"allocated":5140480,"data_stored":2396559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:58 2024","interfaces":[{"interface":"back","average":{"1min":16.789000000000001,"5min":41.939,"15min":24.574999999999999},"min":{"1min":4.0780000000000003,"5min":4.0780000000000003,"15min":4.0780000000000003},"max":{"1min":128.39599999999999,"5min":1631.963,"15min":1631.963},"last":12.478999999999999},{"interface":"front","average":{"1min":14.339,"5min":22.324000000000002,"15min":20.181999999999999},"min":{"1min":6.1740000000000004,"5min":4.4859999999999998,"15min":3.9630000000000001},"max":{"1min":109.867,"5min":138.09800000000001,"15min":370.68799999999999},"last":7.4630000000000001}]},{"osd":1,"last update":"Tue Feb 6 22:17:26 2024","interfaces":[{"interface":"back","average":{"1min":10.836,"5min":52.090000000000003,"15min":32.5},"min":{"1min":6.9000000000000004,"5min":5.9340000000000002,"15min":4.492},"max":{"1min":38.810000000000002,"5min":3146.2530000000002,"15min":3146.2530000000002},"last":6.8440000000000003},{"interface":"front","average":{"1min":19.111000000000001,"5min":58.341999999999999,"15min":36.834000000000003},"min":{"1min":8.4320000000000004,"5min":6.9669999999999996,"15min":6.9020000000000001},"max":{"1min":111.504,"5min":3147.6979999999999,"15min":3147.6979999999999},"last":13.201000000000001}]},{"osd":3,"last update":"Tue Feb 6 22:17:58 2024","interfaces":[{"interface":"back","average":{"1min":12.185,"5min":51.636000000000003,"15min":30.648},"min":{"1min":7.7939999999999996,"5min":7.7939999999999996,"15min":6.1239999999999997},"max":{"1min":18.716999999999999,"5min":2357.8649999999998,"15min":2357.8649999999998},"last":10.189},{"interface":"front","average":{"1min":15.359,"5min":58.621000000000002,"15min":35.381999999999998},"min":{"1min":10.959,"5min":9.6289999999999996,"15min":7.133},"max":{"1min":21.091000000000001,"5min":2367.828,"15min":2367.828},"last":14.894}]},{"osd":4,"last update":"Tue Feb 6 22:17:58 2024","interfaces":[{"interface":"back","average":{"1min":11.063000000000001,"5min":44.146000000000001,"15min":33.569000000000003},"min":{"1min":7.6319999999999997,"5min":7.5949999999999998,"15min":6.5129999999999999},"max":{"1min":15.863,"5min":1805.6099999999999,"15min":1805.6099999999999},"last":8.4619999999999997},{"interface":"front","average":{"1min":15.403,"5min":23.297999999999998,"15min":24.088999999999999},"min":{"1min":9.2959999999999994,"5min":6.976,"15min":6.4009999999999998},"max":{"1min":21.530999999999999,"5min":139.29900000000001,"15min":366.25099999999998},"last":17.594000000000001}]},{"osd":5,"last update":"Tue Feb 6 22:17:58 2024","interfaces":[{"interface":"back","average":{"1min":22.925999999999998,"5min":61.741,"15min":37.472999999999999},"min":{"1min":10.340999999999999,"5min":7.0650000000000004,"15min":6.9610000000000003},"max":{"1min":119.056,"5min":3145.4110000000001,"15min":3145.4110000000001},"last":9.8580000000000005},{"interface":"front","average":{"1min":22.033000000000001,"5min":59.262999999999998,"15min":39.688000000000002},"min":{"1min":12.734,"5min":10.861000000000001,"15min":8.3140000000000001},"max":{"1min":53.290999999999997,"5min":2302.047,"15min":2302.047},"last":16.222999999999999}]},{"osd":6,"last update":"Tue Feb 6 22:17:58 2024","interfaces":[{"interface":"back","average":{"1min":15.878,"5min":31.748000000000001,"15min":28.632000000000001},"min":{"1min":10.866,"5min":7.7039999999999997,"15min":6.6479999999999997},"max":{"1min":24.971,"5min":266.53500000000003,"15min":369.185},"last":12.002000000000001},{"interface":"front","average":{"1min":19.640999999999998,"5min":60.502000000000002,"15min":39.997999999999998},"min":{"1min":13.670999999999999,"5min":12.715,"15min":8.4930000000000003},"max":{"1min":25.183,"5min":2302.8209999999999,"15min":2302.8209999999999},"last":15.544}]},{"osd":7,"last update":"Tue Feb 6 22:17:58 2024","interfaces":[{"interface":"back","average":{"1min":18.395,"5min":28.178000000000001,"15min":24.724},"min":{"1min":9.3499999999999996,"5min":8.7119999999999997,"15min":8.4619999999999997},"max":{"1min":69.673000000000002,"5min":267.28300000000002,"15min":335.39499999999998},"last":14.228999999999999},{"interface":"front","average":{"1min":25.317,"5min":33.137,"15min":28.495000000000001},"min":{"1min":13.324,"5min":11.890000000000001,"15min":10.06},"max":{"1min":151.32499999999999,"5min":268.60899999999998,"15min":342.22699999999998},"last":18.308}]}]},{"osd":7,"up_from":13,"seq":55834575293,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76452,"kb_used_data":5020,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295388,"statfs":{"total":96636764160,"available":96558477312,"internally_reserved":0,"allocated":5140480,"data_stored":2396559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 6 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":11.686999999999999,"5min":23.573,"15min":18.436},"min":{"1min":5.0789999999999997,"5min":4.2880000000000003,"15min":4.2880000000000003},"max":{"1min":28.937000000000001,"5min":283.42899999999997,"15min":283.42899999999997},"last":5.3159999999999998},{"interface":"front","average":{"1min":12.000999999999999,"5min":23.219999999999999,"15min":19.027999999999999},"min":{"1min":5.2949999999999999,"5min":4.3730000000000002,"15min":4.3730000000000002},"max":{"1min":33.189,"5min":279.459,"15min":387.32999999999998},"last":8.7360000000000007}]},{"osd":1,"last update":"Tue Feb 6 22:17:29 2024","interfaces":[{"interface":"back","average":{"1min":13.608000000000001,"5min":19.052,"15min":22.741},"min":{"1min":6.0609999999999999,"5min":6.0609999999999999,"15min":5.577},"max":{"1min":31.280999999999999,"5min":145.70599999999999,"15min":325.41699999999997},"last":11.686},{"interface":"front","average":{"1min":13.199999999999999,"5min":17.952000000000002,"15min":21.225999999999999},"min":{"1min":6.2720000000000002,"5min":6.2720000000000002,"15min":4.7960000000000003},"max":{"1min":30.079999999999998,"5min":159.56100000000001,"15min":380.70800000000003},"last":6.3979999999999997}]},{"osd":2,"last update":"Tue Feb 6 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":15.321999999999999,"5min":78.852000000000004,"15min":42.732999999999997},"min":{"1min":7.1890000000000001,"5min":6.367,"15min":5.8769999999999998},"max":{"1min":56.795000000000002,"5min":3537.9119999999998,"15min":3537.9119999999998},"last":14.484},{"interface":"front","average":{"1min":17,"5min":81.25,"15min":44.497},"min":{"1min":9.3260000000000005,"5min":7.6790000000000003,"15min":6.6539999999999999},"max":{"1min":26.536999999999999,"5min":3538.9259999999999,"15min":3538.9259999999999},"last":10.786}]},{"osd":3,"last update":"Tue Feb 6 22:17:29 2024","interfaces":[{"interface":"back","average":{"1min":16.347999999999999,"5min":20.353999999999999,"15min":21.109999999999999},"min":{"1min":7.7990000000000004,"5min":6.8399999999999999,"15min":6.3390000000000004},"max":{"1min":57.917999999999999,"5min":186.297,"15min":317.94600000000003},"last":15.555},{"interface":"front","average":{"1min":22.106000000000002,"5min":27.684000000000001,"15min":27.027999999999999},"min":{"1min":14.65,"5min":9.8360000000000003,"15min":9.8360000000000003},"max":{"1min":58.795000000000002,"5min":201.75299999999999,"15min":325.60599999999999},"last":16.352}]},{"osd":4,"last update":"Tue Feb 6 22:18:21 2024","interfaces":[{"interface":"back","average":{"1min":18.713999999999999,"5min":22.172000000000001,"15min":20.41},"min":{"1min":9.0129999999999999,"5min":9.0129999999999999,"15min":7.1580000000000004},"max":{"1min":53.256,"5min":102.001,"15min":235.27799999999999},"last":17.187000000000001},{"interface":"front","average":{"1min":16.18,"5min":18.495000000000001,"15min":22.753},"min":{"1min":8.2149999999999999,"5min":6.8440000000000003,"15min":6.8440000000000003},"max":{"1min":39.402000000000001,"5min":87.968999999999994,"15min":350.33300000000003},"last":7.8849999999999998}]},{"osd":5,"last update":"Tue Feb 6 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":24.497,"5min":25.561,"15min":24.407},"min":{"1min":12.058,"5min":7.1189999999999998,"15min":7.1189999999999998},"max":{"1min":89.010999999999996,"5min":201.70400000000001,"15min":214.83500000000001},"last":18.013000000000002},{"interface":"front","average":{"1min":18.481999999999999,"5min":22.195,"15min":25.890999999999998},"min":{"1min":8.6969999999999992,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":65.813999999999993,"5min":106.07299999999999,"15min":301.78399999999999},"last":10.096}]},{"osd":6,"last update":"Tue Feb 6 22:17:29 2024","interfaces":[{"interface":"back","average":{"1min":23.632999999999999,"5min":26.312999999999999,"15min":25.606000000000002},"min":{"1min":15.241,"5min":8.3239999999999998,"15min":8.3239999999999998},"max":{"1min":60.616,"5min":203.58199999999999,"15min":235.27799999999999},"last":18.937000000000001},{"interface":"front","average":{"1min":16.300999999999998,"5min":22.77,"15min":26.331},"min":{"1min":9.3000000000000007,"5min":7.3179999999999996,"15min":6.7400000000000002},"max":{"1min":28.381,"5min":185.49799999999999,"15min":490.68299999999999},"last":9.4879999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575290,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72292,"kb_used_data":5020,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299548,"statfs":{"total":96636764160,"available":96562737152,"internally_reserved":0,"allocated":5140480,"data_stored":2396559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 6 22:17:53 2024","interfaces":[{"interface":"back","average":{"1min":15.26,"5min":28.606999999999999,"15min":23.428000000000001},"min":{"1min":5.6970000000000001,"5min":4.633,"15min":4.4029999999999996},"max":{"1min":98.292000000000002,"5min":314.11000000000001,"15min":416.55599999999998},"last":9.1720000000000006},{"interface":"front","average":{"1min":13.664999999999999,"5min":22.065999999999999,"15min":21.591000000000001},"min":{"1min":6.5339999999999998,"5min":5.5389999999999997,"15min":4.5},"max":{"1min":81.653999999999996,"5min":199.238,"15min":407.44799999999998},"last":8.4830000000000005}]},{"osd":2,"last update":"Tue Feb 6 22:17:53 2024","interfaces":[{"interface":"back","average":{"1min":17.803999999999998,"5min":28.433,"15min":27.036999999999999},"min":{"1min":7.2770000000000001,"5min":6.4880000000000004,"15min":6.3799999999999999},"max":{"1min":93.007999999999996,"5min":264.48200000000003,"15min":395.14800000000002},"last":13.475},{"interface":"front","average":{"1min":15.856,"5min":29.306000000000001,"15min":25.323},"min":{"1min":5.9100000000000001,"5min":5.5259999999999998,"15min":5.383},"max":{"1min":99.483000000000004,"5min":514.94500000000005,"15min":514.94500000000005},"last":10.617000000000001}]},{"osd":3,"last update":"Tue Feb 6 22:17:53 2024","interfaces":[{"interface":"back","average":{"1min":19.762,"5min":26.550000000000001,"15min":24.818000000000001},"min":{"1min":8.6289999999999996,"5min":7.2409999999999997,"15min":5.8140000000000001},"max":{"1min":105.68000000000001,"5min":291.59100000000001,"15min":376.95100000000002},"last":9.7949999999999999},{"interface":"front","average":{"1min":14.304,"5min":25.513000000000002,"15min":25.059999999999999},"min":{"1min":7.75,"5min":6.6139999999999999,"15min":6.6139999999999999},"max":{"1min":80.340000000000003,"5min":262.06900000000002,"15min":395.83100000000002},"last":16.713000000000001}]},{"osd":4,"last update":"Tue Feb 6 22:18:09 2024","interfaces":[{"interface":"back","average":{"1min":23.501999999999999,"5min":32.116,"15min":27.870999999999999},"min":{"1min":10.741,"5min":8.8350000000000009,"15min":6.8540000000000001},"max":{"1min":103.437,"5min":491.53100000000001,"15min":491.53100000000001},"last":19.620000000000001},{"interface":"front","average":{"1min":18.687999999999999,"5min":27.125,"15min":24.09},"min":{"1min":8.6620000000000008,"5min":7.2409999999999997,"15min":6.6589999999999998},"max":{"1min":100.51900000000001,"5min":490.28699999999998,"15min":490.28699999999998},"last":14.731999999999999}]},{"osd":5,"last update":"Tue Feb 6 22:17:53 2024","interfaces":[{"interface":"back","average":{"1min":22.463999999999999,"5min":31.809999999999999,"15min":28.338000000000001},"min":{"1min":10.941000000000001,"5min":7.9969999999999999,"15min":7.9969999999999999},"max":{"1min":104.276,"5min":282.00799999999998,"15min":480.24200000000002},"last":12.611000000000001},{"interface":"front","average":{"1min":22.041,"5min":27.257000000000001,"15min":23.969000000000001},"min":{"1min":9.7750000000000004,"5min":8.9350000000000005,"15min":7.6059999999999999},"max":{"1min":94.417000000000002,"5min":193.21700000000001,"15min":347.78699999999998},"last":11.81}]},{"osd":6,"last update":"Tue Feb 6 22:17:53 2024","interfaces":[{"interface":"back","average":{"1min":24.131,"5min":32.994999999999997,"15min":30.001999999999999},"min":{"1min":12.452,"5min":9.4789999999999992,"15min":8.9139999999999997},"max":{"1min":106.277,"5min":218.12,"15min":550.89700000000005},"last":15.967000000000001},{"interface":"front","average":{"1min":20.716000000000001,"5min":27.524999999999999,"15min":25.503},"min":{"1min":10.202,"5min":9.1310000000000002,"15min":8.4730000000000008},"max":{"1min":96.307000000000002,"5min":175.947,"15min":486.25299999999999},"last":14.084}]},{"osd":7,"last update":"Tue Feb 6 22:17:38 2024","interfaces":[{"interface":"back","average":{"1min":24.734000000000002,"5min":33.390999999999998,"15min":30.097000000000001},"min":{"1min":13.772,"5min":11.374000000000001,"15min":9.6170000000000009},"max":{"1min":107.01000000000001,"5min":218.108,"15min":404.22800000000001},"last":15.401999999999999},{"interface":"front","average":{"1min":28.641999999999999,"5min":38.777999999999999,"15min":34.070999999999998},"min":{"1min":15.483000000000001,"5min":13.800000000000001,"15min":11.497999999999999},"max":{"1min":109.504,"5min":219.56299999999999,"15min":415.92000000000002},"last":17.73}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-06T22:18:28.645 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-02-06T22:18:28.748 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.747+0000 7f1ad4fcf640 10 monclient: get_monmap_and_config 2024-02-06T22:18:28.748 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.747+0000 7f1ad4fcf640 10 monclient: build_initial_monmap 2024-02-06T22:18:28.748 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.747+0000 7f1ad4fcf640 10 monclient: monmap: 2024-02-06T22:18:28.748 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:28.748 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:28.748 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T22:18:28.748380+0000 2024-02-06T22:18:28.749 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T22:18:28.748380+0000 2024-02-06T22:18:28.749 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:28.749 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:28.749 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.120:6789/0 mon.noname-a-legacy 2024-02-06T22:18:28.749 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2024-02-06T22:18:28.749 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-b 2024-02-06T22:18:28.749 INFO:teuthology.orchestra.run.smithi120.stderr:3: v2:172.21.15.120:3300/0 mon.noname-a 2024-02-06T22:18:28.749 INFO:teuthology.orchestra.run.smithi120.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2024-02-06T22:18:28.749 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:28.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.747+0000 7f1ad4fcf640 1 Processor -- start 2024-02-06T22:18:28.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.747+0000 7f1ad4fcf640 1 -- start start 2024-02-06T22:18:28.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.747+0000 7f1ad4fcf640 10 monclient: init 2024-02-06T22:18:28.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.748+0000 7f1ad4fcf640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:28.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.748+0000 7f1ad4fcf640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-02-06T22:18:28.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.748+0000 7f1ad4fcf640 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1ad00b54b0 0x7f1ad00a9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:28.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.748+0000 7f1ad4fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f1ad00b54b0 addr v2:172.21.15.135:3300/0 2024-02-06T22:18:28.751 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.748+0000 7f1ad4fcf640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1ad013f830 0x7f1ad013fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:28.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.748+0000 7f1ad4fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f1ad013f830 addr v2:172.21.15.120:3300/0 2024-02-06T22:18:28.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.748+0000 7f1ad4fcf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ad0142680 addr v1:172.21.15.135:6789/0 2024-02-06T22:18:28.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.748+0000 7f1ad4fcf640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad0143200 con 0x7f1ad0142680 2024-02-06T22:18:28.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.748+0000 7f1ad4fcf640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:28.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.748+0000 7f1ad4fcf640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad00ab520 con 0x7f1ad013f830 2024-02-06T22:18:28.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.748+0000 7f1ad4fcf640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:28.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.748+0000 7f1ad4fcf640 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad00b5890 con 0x7f1ad00b54b0 2024-02-06T22:18:28.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.748+0000 7f1ad4fcf640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:28.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.748+0000 7f1ad4fcf640 10 monclient(hunting): authenticate will time out at 3539.086019s 2024-02-06T22:18:28.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.749+0000 7f1acdd9b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1ad013f830 0x7f1ad013fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:28.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.749+0000 7f1acdd9b640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1ad013f830 0x7f1ad013fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:53366/0 (socket says 172.21.15.120:53366) 2024-02-06T22:18:28.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.749+0000 7f1acdd9b640 1 -- 172.21.15.120:0/284327527 learned_addr learned my addr 172.21.15.120:0/284327527 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:28.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.749+0000 7f1acdd9b640 10 monclient(hunting): get_auth_request con 0x7f1ad013f830 auth_method 0 2024-02-06T22:18:28.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.749+0000 7f1acdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:28.752 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.749+0000 7f1acdd9b640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:28.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.749+0000 7f1acdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:28.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.749+0000 7f1ace59c640 1 --2- 172.21.15.120:0/284327527 >> v2:172.21.15.135:3300/0 conn(0x7f1ad00b54b0 0x7f1ad00a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:28.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.749+0000 7f1ace59c640 10 monclient(hunting): get_auth_request con 0x7f1ad00b54b0 auth_method 0 2024-02-06T22:18:28.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.749+0000 7f1ace59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:28.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.749+0000 7f1ace59c640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:28.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.749+0000 7f1ace59c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:28.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.749+0000 7f1acdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:28.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.749+0000 7f1acdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:28.753 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.749+0000 7f1acdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:28.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.750+0000 7f1ace59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:28.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.750+0000 7f1ace59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:28.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.750+0000 7f1ace59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:28.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.751+0000 7f1acdd9b640 10 monclient(hunting): handle_auth_done global_id 16608 payload 274 2024-02-06T22:18:28.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.751+0000 7f1acdd9b640 1 -- 172.21.15.120:0/284327527 >> v2:172.21.15.135:3300/0 conn(0x7f1ad00b54b0 msgr2=0x7f1ad00a9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-06T22:18:28.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.751+0000 7f1acdd9b640 1 --2- 172.21.15.120:0/284327527 >> v2:172.21.15.135:3300/0 conn(0x7f1ad00b54b0 0x7f1ad00a9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.754 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.751+0000 7f1acdd9b640 1 -- 172.21.15.120:0/284327527 >> v1:172.21.15.135:6789/0 conn(0x7f1ad0142680 legacy=0x7f1ad0142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:28.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.751+0000 7f1acdd9b640 10 monclient: _finish_hunting 0 2024-02-06T22:18:28.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.751+0000 7f1acdd9b640 1 monclient: found mon.noname-a 2024-02-06T22:18:28.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.751+0000 7f1acdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:28.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.751+0000 7f1acdd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:18:28.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.751+0000 7f1acdd9b640 1 -- 172.21.15.120:0/284327527 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad00ab720 con 0x7f1ad013f830 2024-02-06T22:18:28.755 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.753+0000 7f1acdd9b640 1 --2- 172.21.15.120:0/284327527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad013f830 0x7f1ad013fc70 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7f1ad006c7b0 tx=0x7f1ac402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a075ddc4b67e5d server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:28.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1acd59a640 1 -- 172.21.15.120:0/284327527 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4030860 con 0x7f1ad013f830 2024-02-06T22:18:28.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1acd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:28.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1acd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:18:28.758 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1acd59a640 10 monclient: dump: 2024-02-06T22:18:28.758 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:28.758 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:28.758 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.758 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1acd59a640 10 monclient: _finish_auth 0 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1acd59a640 10 monclient: _check_auth_tickets 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1acd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1acd59a640 1 -- 172.21.15.120:0/284327527 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac40309c0 con 0x7f1ad013f830 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1acd59a640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1acd59a640 1 -- 172.21.15.120:0/284327527 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4030d50 con 0x7f1ad013f830 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1ad4fcf640 5 monclient: authenticate success, global_id 16608 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1ad4fcf640 10 monclient: get_monmap_and_config success 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1ad4fcf640 4 set_mon_vals no callback set 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1acd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:28.759 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1acd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:18:28.760 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.756+0000 7f1acd59a640 10 monclient: dump: 2024-02-06T22:18:28.760 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:28.760 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:28.760 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.760 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.760 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:28.760 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:28.760 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:28.760 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:28.760 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:28.760 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:28.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.757+0000 7f1ad4fcf640 10 monclient: shutdown 2024-02-06T22:18:28.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.757+0000 7f1ad4fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:28.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.757+0000 7f1ad4fcf640 1 -- 172.21.15.120:0/284327527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad013f830 msgr2=0x7f1ad013fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:28.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.757+0000 7f1ad4fcf640 1 --2- 172.21.15.120:0/284327527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad013f830 0x7f1ad013fc70 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7f1ad006c7b0 tx=0x7f1ac402b9a0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.757+0000 7f1ad4fcf640 1 -- 172.21.15.120:0/284327527 shutdown_connections 2024-02-06T22:18:28.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.757+0000 7f1ad4fcf640 1 --2- 172.21.15.120:0/284327527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad013f830 0x7f1ad013fc70 unknown :-1 s=CLOSED pgs=3369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.757+0000 7f1ad4fcf640 1 --2- 172.21.15.120:0/284327527 >> v2:172.21.15.135:3300/0 conn(0x7f1ad00b54b0 0x7f1ad00a9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.763 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.757+0000 7f1ad4fcf640 1 -- 172.21.15.120:0/284327527 >> 172.21.15.120:0/284327527 conn(0x7f1ad00b8660 msgr2=0x7f1ad00b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:28.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.757+0000 7f1ad4fcf640 1 -- 172.21.15.120:0/284327527 shutdown_connections 2024-02-06T22:18:28.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.757+0000 7f1ad4fcf640 1 -- 172.21.15.120:0/284327527 wait complete. 2024-02-06T22:18:28.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.758+0000 7f1ad4fcf640 10 monclient: build_initial_monmap 2024-02-06T22:18:28.764 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.758+0000 7f1ad4fcf640 10 monclient: monmap: 2024-02-06T22:18:28.764 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-02-06T22:18:28.764 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-02-06T22:18:28.764 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-02-06T22:18:28.764 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-02-06T22:18:28.764 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-02-06T22:18:28.764 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:28.764 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.noname-a 2024-02-06T22:18:28.764 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2024-02-06T22:18:28.764 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.noname-c 2024-02-06T22:18:28.764 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:28.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.758+0000 7f1ad4fcf640 1 librados: starting msgr at 2024-02-06T22:18:28.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.758+0000 7f1ad4fcf640 1 librados: starting objecter 2024-02-06T22:18:28.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.758+0000 7f1ad4fcf640 1 Processor -- start 2024-02-06T22:18:28.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.758+0000 7f1ad4fcf640 1 -- start start 2024-02-06T22:18:28.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.758+0000 7f1ad4fcf640 1 librados: setting wanted keys 2024-02-06T22:18:28.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.758+0000 7f1ad4fcf640 1 librados: calling monclient init 2024-02-06T22:18:28.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.758+0000 7f1ad4fcf640 10 monclient: init 2024-02-06T22:18:28.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 10 monclient: _reopen_session rank -1 2024-02-06T22:18:28.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-02-06T22:18:28.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad00b54b0 0x7f1ad00b79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:28.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f1ad00b54b0 addr [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-02-06T22:18:28.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ad013f830 0x7f1ad0136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:28.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f1ad013f830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-02-06T22:18:28.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ad0142680 0x7f1ad01387d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:28.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f1ad0142680 addr [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad013c220 con 0x7f1ad00b54b0 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad014cab0 con 0x7f1ad013f830 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 10 monclient(hunting): start opening mon connection 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad0136560 con 0x7f1ad0142680 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 10 monclient(hunting): _renew_subs 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ad4fcf640 10 monclient(hunting): authenticate will time out at 3539.097042s 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.759+0000 7f1ace59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad00b54b0 0x7f1ad00b79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1aced9d640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ad0142680 0x7f1ad01387d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1acdd9b640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ad013f830 0x7f1ad0136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1ace59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad00b54b0 0x7f1ad00b79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:53376/0 (socket says 172.21.15.120:53376) 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1ace59c640 1 -- 172.21.15.120:0/3918906026 learned_addr learned my addr 172.21.15.120:0/3918906026 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1ace59c640 10 monclient(hunting): get_auth_request con 0x7f1ad00b54b0 auth_method 0 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1ace59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:28.766 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1ace59c640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1ace59c640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1acdd9b640 10 monclient(hunting): get_auth_request con 0x7f1ad013f830 auth_method 0 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1acdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1acdd9b640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1acdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1aced9d640 10 monclient(hunting): get_auth_request con 0x7f1ad0142680 auth_method 0 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1aced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1aced9d640 10 monclient(hunting): _init_auth method 2 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1aced9d640 10 monclient(hunting): _init_auth creating new auth 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1ace59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1ace59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1ace59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1acdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1acdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.760+0000 7f1acdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:28.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.761+0000 7f1aced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-02-06T22:18:28.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.761+0000 7f1aced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-02-06T22:18:28.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.761+0000 7f1aced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-02-06T22:18:28.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.763+0000 7f1ace59c640 10 monclient(hunting): handle_auth_done global_id 16611 payload 824 2024-02-06T22:18:28.768 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.763+0000 7f1ace59c640 1 -- 172.21.15.120:0/3918906026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ad0142680 msgr2=0x7f1ad01387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:28.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.763+0000 7f1ace59c640 1 --2- 172.21.15.120:0/3918906026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ad0142680 0x7f1ad01387d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.763+0000 7f1ace59c640 1 -- 172.21.15.120:0/3918906026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ad013f830 msgr2=0x7f1ad0136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:28.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.763+0000 7f1ace59c640 1 --2- 172.21.15.120:0/3918906026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ad013f830 0x7f1ad0136dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:28.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.763+0000 7f1ace59c640 10 monclient: _finish_hunting 0 2024-02-06T22:18:28.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.763+0000 7f1ace59c640 1 monclient: found mon.noname-a 2024-02-06T22:18:28.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.763+0000 7f1ace59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-02-06T22:18:28.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.763+0000 7f1ace59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.120:3300/0 2024-02-06T22:18:28.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.763+0000 7f1ace59c640 1 -- 172.21.15.120:0/3918906026 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac40094d0 con 0x7f1ad00b54b0 2024-02-06T22:18:28.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.765+0000 7f1ace59c640 1 --2- 172.21.15.120:0/3918906026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad00b54b0 0x7f1ad00b79c0 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7f1ac000efc0 tx=0x7f1ac000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:28.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.765+0000 7f1abeffd640 10 client.?.objecter ms_handle_connect 0x7f1ad00b54b0 2024-02-06T22:18:28.771 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.765+0000 7f1abeffd640 10 client.?.objecter resend_mon_ops 2024-02-06T22:18:28.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1abeffd640 1 -- 172.21.15.120:0/3918906026 <== mon.0 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0014480 con 0x7f1ad00b54b0 2024-02-06T22:18:28.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1abeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:28.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1abeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-02-06T22:18:28.773 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1abeffd640 10 monclient: dump: 2024-02-06T22:18:28.773 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:28.773 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:28.773 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.773 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.773 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:28.773 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:28.773 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:28.773 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:28.773 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:28.773 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1abeffd640 10 monclient: _finish_auth 0 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1abeffd640 10 monclient: _check_auth_tickets 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1abeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1ad4fcf640 5 monclient: authenticate success, global_id 16611 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1ad4fcf640 10 monclient: _renew_subs 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1ad4fcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1ad4fcf640 1 -- 172.21.15.120:0/3918906026 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad013c6b0 con 0x7f1ad00b54b0 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1ad4fcf640 10 client.16611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1ad4fcf640 10 monclient: _renew_subs 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1ad4fcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1ad4fcf640 1 -- 172.21.15.120:0/3918906026 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad0138ef0 con 0x7f1ad00b54b0 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.767+0000 7f1ad4fcf640 1 librados: init done 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.768+0000 7f1ad4fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.768+0000 7f1ad4fcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.768+0000 7f1ad4fcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:28.774 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.768+0000 7f1ad4fcf640 1 -- 172.21.15.120:0/3918906026 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad004ef90 con 0x7f1ad00b54b0 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.769+0000 7f1abeffd640 1 -- 172.21.15.120:0/3918906026 <== mon.0 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac000f040 con 0x7f1ad00b54b0 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.769+0000 7f1abeffd640 10 monclient: handle_config config(0 keys) v1 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.769+0000 7f1abeffd640 1 -- 172.21.15.120:0/3918906026 <== mon.0 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0014800 con 0x7f1ad00b54b0 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.769+0000 7f1abeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.769+0000 7f1abeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.769+0000 7f1abeffd640 10 monclient: dump: 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 4e4791bf-0a02-4478-9517-2d2e893f4f7c 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-02-06T21:39:47.348445+0000 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:0: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.a 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:18:28.775 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:28.769+0000 7f1accd99640 4 set_mon_vals no callback set 2024-02-06T22:18:29.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.019+0000 7f1abeffd640 1 -- 172.21.15.120:0/3918906026 <== mon.0 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1ac0014960 con 0x7f1ad00b54b0 2024-02-06T22:18:29.021 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.020+0000 7f1abeffd640 1 --2- 172.21.15.120:0/3918906026 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1ab006d1a0 0x7f1ab006f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-06T22:18:29.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.020+0000 7f1acdd9b640 1 --2- 172.21.15.120:0/3918906026 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1ab006d1a0 0x7f1ab006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-06T22:18:29.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.020+0000 7f1acdd9b640 10 monclient: get_auth_request con 0x7f1ab006d1a0 auth_method 0 2024-02-06T22:18:29.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.021+0000 7f1acdd9b640 1 --2- 172.21.15.120:0/3918906026 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1ab006d1a0 0x7f1ab006f660 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4036000 tx=0x7f1ac4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-06T22:18:29.022 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.021+0000 7f1abeffd640 10 client.16611.objecter ms_handle_connect 0x7f1ab006d1a0 2024-02-06T22:18:29.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.032+0000 7f1abeffd640 1 -- 172.21.15.120:0/3918906026 <== mon.0 v2:172.21.15.120:3300/0 5 ==== osd_map(1346..1346 src has 1047..1346) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f1ac0059c40 con 0x7f1ad00b54b0 2024-02-06T22:18:29.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.032+0000 7f1abeffd640 10 client.16611.objecter ms_dispatch 0x7f1ad0000b90 osd_map(1346..1346 src has 1047..1346) v4 2024-02-06T22:18:29.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.032+0000 7f1abeffd640 3 client.16611.objecter handle_osd_map got epochs [1346,1346] > 0 2024-02-06T22:18:29.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.032+0000 7f1abeffd640 3 client.16611.objecter handle_osd_map decoding full epoch 1346 2024-02-06T22:18:29.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.033+0000 7f1abeffd640 20 client.16611.objecter dump_active .. 0 homeless 2024-02-06T22:18:29.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.033+0000 7f1abeffd640 1 -- 172.21.15.120:0/3918906026 <== mon.0 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1ac0055ee0 con 0x7f1ad00b54b0 2024-02-06T22:18:29.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.033+0000 7f1abeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-02-06T22:18:29.035 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.033+0000 7f1abeffd640 10 monclient: _finish_command 1 = system:0 2024-02-06T22:18:29.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.225+0000 7f1ad4fcf640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-02-06T22:18:29.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.225+0000 7f1ad4fcf640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-02-06T22:18:29.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.225+0000 7f1ad4fcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.120:3300/0 2024-02-06T22:18:29.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.225+0000 7f1ad4fcf640 1 -- 172.21.15.120:0/3918906026 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f1ad0067e10 con 0x7f1ad00b54b0 2024-02-06T22:18:29.238 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:09.052 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:29.246 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:09.061 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:29.268 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:09.083 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:29.295 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:09.059 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:29.298 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:09.063 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:29.302 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:09.117 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:29.304 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:09.119 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:29.314 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:09.078 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:29.318 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:09.133 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:29.427 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.426+0000 7f1abeffd640 1 -- 172.21.15.120:0/3918906026 <== mon.0 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v8)=0 v8) v1 ==== 123+0+0 (secure 0 0 0) 0x7f1ac00180c0 con 0x7f1ad00b54b0 2024-02-06T22:18:29.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.426+0000 7f1abeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-02-06T22:18:29.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.426+0000 7f1abeffd640 10 monclient: _finish_command 2 = system:0 2024-02-06T22:18:29.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.426+0000 7f1abeffd640 1 -- 172.21.15.120:0/3918906026 <== mon.0 v2:172.21.15.120:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f1ac0055ee0 con 0x7f1ad00b54b0 2024-02-06T22:18:29.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.426+0000 7f1abeffd640 10 monclient: handle_config config(1 keys) v1 2024-02-06T22:18:29.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.426+0000 7f1accd99640 4 set_mon_vals no callback set 2024-02-06T22:18:29.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.426+0000 7f1accd99640 10 set_mon_vals mon_health_to_clog = false 2024-02-06T22:18:29.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 10 librados: watch_flush enter 2024-02-06T22:18:29.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 10 librados: watch_flush exit 2024-02-06T22:18:29.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 20 client.16611.objecter shutdown clearing up homeless session... 2024-02-06T22:18:29.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 10 client.16611.objecter successfully canceled tick 2024-02-06T22:18:29.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 1 -- 172.21.15.120:0/3918906026 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1ab006d1a0 msgr2=0x7f1ab006f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:29.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 1 --2- 172.21.15.120:0/3918906026 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1ab006d1a0 0x7f1ab006f660 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4036000 tx=0x7f1ac4008030 comp rx=0 tx=0).stop 2024-02-06T22:18:29.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 10 monclient: shutdown 2024-02-06T22:18:29.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-02-06T22:18:29.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 1 -- 172.21.15.120:0/3918906026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad00b54b0 msgr2=0x7f1ad00b79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-06T22:18:29.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 1 --2- 172.21.15.120:0/3918906026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad00b54b0 0x7f1ad00b79c0 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7f1ac000efc0 tx=0x7f1ac000c490 comp rx=0 tx=0).stop 2024-02-06T22:18:29.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 1 -- 172.21.15.120:0/3918906026 shutdown_connections 2024-02-06T22:18:29.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 1 --2- 172.21.15.120:0/3918906026 >> [v2:172.21.15.135:6800/315664144,v1:172.21.15.135:6801/315664144] conn(0x7f1ab006d1a0 0x7f1ab006f660 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:29.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 1 --2- 172.21.15.120:0/3918906026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ad013f830 0x7f1ad0136dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:29.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 1 --2- 172.21.15.120:0/3918906026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ad0142680 0x7f1ad01387d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:29.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 1 --2- 172.21.15.120:0/3918906026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ad00b54b0 0x7f1ad00b79c0 unknown :-1 s=CLOSED pgs=3370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-06T22:18:29.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 1 -- 172.21.15.120:0/3918906026 >> 172.21.15.120:0/3918906026 conn(0x7f1ad00b8660 msgr2=0x7f1ad00b3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-06T22:18:29.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 1 -- 172.21.15.120:0/3918906026 shutdown_connections 2024-02-06T22:18:29.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 1 -- 172.21.15.120:0/3918906026 wait complete. 2024-02-06T22:18:29.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-02-06T22:18:29.429+0000 7f1ad4fcf640 1 librados: shutdown 2024-02-06T22:18:29.445 INFO:teuthology.misc:Shutting down mds daemons... 2024-02-06T22:18:29.445 INFO:teuthology.misc:Shutting down osd daemons... 2024-02-06T22:18:29.445 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-02-06T22:18:29.445 INFO:teuthology.orchestra.run:waiting for 300 2024-02-06T22:18:29.482 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T22:18:29.467+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 34806) UID: 0 2024-02-06T22:18:29.482 INFO:tasks.ceph.osd.0.smithi120.stderr:2024-02-06T22:18:29.470+0000 b47a640 -1 osd.0 1346 *** Got signal Terminated *** 2024-02-06T22:18:39.054 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:18.869 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:39.155 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:18.918 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:39.155 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:18.939 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:39.155 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:18.867 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:39.155 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:18.874 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:39.155 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:18.892 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:39.155 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:18.970 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:39.158 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:18.973 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:39.176 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:18.991 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:44.285 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:24.100 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:44.288 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:24.103 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:44.306 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:24.121 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:44.330 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:24.095 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:44.336 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:24.098 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:44.338 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:24.153 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:44.343 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:24.158 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:44.346 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:24.110 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:44.356 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:24.171 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:59.232 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:39.047 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:59.235 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:39.050 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:59.255 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:39.070 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:59.283 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:39.098 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:59.287 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:39.102 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:59.289 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:39.053 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:59.290 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:39.055 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:59.305 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:39.120 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:18:59.318 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:39.082 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:04.244 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:44.059 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:04.248 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:44.063 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:04.262 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:44.076 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:04.294 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:44.109 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:04.295 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:44.059 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:04.297 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:44.062 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:04.298 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:44.113 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:04.308 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:44.072 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:04.309 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:44.124 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:19.287 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:59.102 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:19.291 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:59.106 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:19.307 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:38:59.122 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:19.338 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:59.102 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:19.342 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:59.106 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:19.356 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:38:59.120 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:19.357 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:59.172 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:19.359 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:59.174 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:19.380 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:38:59.195 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:24.297 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:04.112 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:24.301 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:04.065 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:24.301 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:04.116 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:24.303 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:04.067 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:24.314 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:04.078 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:24.319 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:04.134 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:24.356 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:04.170 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:24.358 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:04.173 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:24.373 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:04.188 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:28.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-02-06T22:19:30.458 INFO:tasks.ceph.osd.0:Stopped 2024-02-06T22:19:30.458 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-02-06T22:19:30.459 INFO:teuthology.orchestra.run:waiting for 300 2024-02-06T22:19:30.471 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T22:19:30.462+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 34820) UID: 0 2024-02-06T22:19:30.472 INFO:tasks.ceph.osd.1.smithi120.stderr:2024-02-06T22:19:30.464+0000 b47a640 -1 osd.1 1352 *** Got signal Terminated *** 2024-02-06T22:19:34.328 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:14.143 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:34.331 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:14.146 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:34.346 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:14.161 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:34.376 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:14.140 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:34.379 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:14.143 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:34.379 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:14.194 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:34.385 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:14.200 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:34.393 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:14.158 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:34.399 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:14.214 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:34.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-02-06T22:19:34.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-02-06T22:19:36.561 INFO:tasks.ceph.osd.1:Stopped 2024-02-06T22:19:36.561 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-02-06T22:19:36.561 INFO:teuthology.orchestra.run:waiting for 300 2024-02-06T22:19:36.573 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T22:19:36.565+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 34819) UID: 0 2024-02-06T22:19:36.573 INFO:tasks.ceph.osd.2.smithi120.stderr:2024-02-06T22:19:36.566+0000 b47a640 -1 osd.2 1354 *** Got signal Terminated *** 2024-02-06T22:19:40.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-02-06T22:19:44.510 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:24.325 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:44.513 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:24.328 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:44.529 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:24.344 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:44.561 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:24.325 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:44.562 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:24.327 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:44.564 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:24.379 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:44.565 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:24.380 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:44.577 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:24.341 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:44.581 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:24.396 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:54.419 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:34.233 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:54.422 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:34.237 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:54.442 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:34.257 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:54.474 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:34.238 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:54.474 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:34.289 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:54.477 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:34.291 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:54.477 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:34.241 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:54.489 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:34.304 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:19:54.489 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:34.253 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:04.448 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:44.263 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:04.450 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:44.265 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:04.466 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:44.281 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:04.499 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:44.314 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:04.500 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:44.264 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:04.502 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:44.317 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:04.503 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:44.267 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:04.515 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:44.280 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:04.518 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:44.333 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:14.424 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:54.239 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:14.472 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:54.287 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:14.491 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:39:54.306 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:14.520 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:54.335 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:14.520 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:54.284 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:14.522 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:54.337 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:14.522 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:54.287 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:14.534 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:39:54.349 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:14.534 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:39:54.299 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:24.580 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:04.395 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:24.583 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:04.399 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:24.603 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:04.418 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:24.628 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:04.392 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:24.631 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:04.396 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:24.634 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:04.449 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:24.637 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:04.452 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:24.644 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:04.408 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:24.650 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:04.465 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:34.557 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:14.372 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:34.559 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:14.374 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:34.573 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:14.388 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:34.608 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:14.372 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:34.610 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:14.375 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:34.622 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:14.386 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:34.627 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:14.442 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:34.628 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:14.443 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:34.639 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:14.454 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:49.581 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:29.396 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:49.631 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:29.446 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:49.648 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:29.463 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:49.681 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:29.445 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:49.683 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:29.498 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:49.684 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:29.499 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:49.685 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:29.450 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:49.700 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:29.515 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:49.701 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:29.465 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:54.581 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:34.396 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:54.629 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:34.393 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:54.629 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:34.445 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:54.634 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:34.398 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:54.647 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:34.411 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:54.651 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:34.466 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:54.681 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:34.495 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:54.685 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:34.500 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:20:54.699 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:34.513 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:09.618 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:49.433 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:09.667 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:49.481 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:09.672 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:49.436 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:09.676 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:49.440 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:09.696 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:49.460 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:09.699 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:49.514 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:09.727 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:49.542 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:09.728 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:49.543 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:09.748 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:49.562 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:14.678 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:54.493 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:14.683 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:54.498 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:14.699 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:40:54.514 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:14.729 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:54.544 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:14.731 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:54.495 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:14.733 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:54.497 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:14.733 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:54.548 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:14.745 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:40:54.509 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:14.746 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:40:54.561 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:29.715 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:09.530 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:29.720 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:09.535 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:29.737 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:09.552 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:29.766 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:09.581 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:29.771 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:09.586 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:29.772 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:09.536 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:29.774 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:09.539 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:29.791 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:09.555 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:29.812 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:09.626 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:30.530 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:10.294 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:30.531 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:10.296 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:30.543 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:10.308 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:30.594 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:10.409 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:30.595 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:10.410 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:30.613 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:10.428 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:30.618 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:10.433 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:30.619 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:10.434 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:30.631 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:10.446 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:34.733 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:14.548 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:34.738 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:14.553 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:34.754 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:14.569 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:34.791 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:14.555 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:34.795 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:14.559 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:34.807 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:14.571 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:34.808 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:14.623 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:34.810 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:14.625 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:34.820 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:14.635 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:49.786 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:29.601 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:49.788 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:29.603 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:49.804 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:29.619 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:49.836 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:29.600 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:49.838 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:29.602 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:49.839 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:29.654 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:49.841 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:29.657 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:49.856 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:29.671 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:49.856 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:29.621 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:54.750 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:34.565 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:54.798 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:34.613 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:54.816 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:34.631 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:54.844 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:34.659 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:54.845 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:34.609 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:54.848 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:34.612 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:54.852 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:34.667 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:54.860 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:34.624 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:21:54.863 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:34.678 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:09.838 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:49.653 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:09.841 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:49.656 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:09.858 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:49.673 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:09.893 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:49.657 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:09.895 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:49.659 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:09.907 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:49.721 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:09.910 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:49.725 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:09.911 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:49.675 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:09.933 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:49.747 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:14.854 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:54.668 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:14.854 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:54.618 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:14.855 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:54.670 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:14.857 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:54.621 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:14.871 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:41:54.635 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:14.874 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:41:54.689 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:14.907 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:54.721 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:14.912 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:54.727 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:14.926 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:41:54.741 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:19.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-02-06T22:22:21.298 INFO:tasks.ceph.osd.2:Stopped 2024-02-06T22:22:21.299 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-02-06T22:22:21.299 INFO:teuthology.orchestra.run:waiting for 300 2024-02-06T22:22:21.311 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T22:22:21.302+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 34818) UID: 0 2024-02-06T22:22:21.313 INFO:tasks.ceph.osd.3.smithi120.stderr:2024-02-06T22:22:21.303+0000 b47a640 -1 osd.3 1358 *** Got signal Terminated *** 2024-02-06T22:22:25.297 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-02-06T22:22:29.897 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:09.712 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:29.900 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:09.715 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:29.926 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:09.741 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:29.947 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:09.762 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:29.949 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:09.764 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:29.950 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:09.715 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:29.953 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:09.717 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:29.969 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:09.784 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:29.972 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:09.736 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:30.592 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:10.357 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:30.595 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:10.358 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:30.601 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:10.366 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:30.671 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:10.486 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:30.672 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:10.487 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:30.676 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:10.491 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:30.677 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:10.492 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:30.681 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:10.496 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:30.686 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:10.501 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:49.950 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:29.764 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:49.954 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:29.769 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:49.975 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:29.790 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:50.003 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:29.767 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:50.005 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:29.770 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:50.018 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:29.833 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:50.023 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:29.787 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:50.024 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:29.840 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:50.043 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:29.858 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:50.626 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:30.390 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:50.627 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:30.392 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:50.639 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:30.403 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:50.674 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:30.489 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:50.675 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:30.490 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:50.683 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:30.498 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:50.711 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:30.526 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:50.712 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:30.527 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:22:50.720 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:30.535 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.001 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:49.815 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.007 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:49.822 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.027 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:49.842 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.052 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:49.867 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.055 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:49.820 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.057 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:49.872 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.057 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:49.821 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.076 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:49.840 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.083 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:49.898 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.696 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:50.460 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.697 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:50.461 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.705 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:42:50.469 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.726 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:50.541 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.727 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:50.542 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.738 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:42:50.553 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.766 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:50.581 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.767 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:50.582 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:10.775 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:42:50.590 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.067 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:09.882 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.070 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:09.885 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.086 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:09.901 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.115 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:09.930 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.118 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:09.933 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.122 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:09.886 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.123 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:09.888 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.138 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:09.953 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.140 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:09.905 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.757 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:10.521 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.758 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:10.522 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.766 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:10.530 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.822 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:10.636 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.822 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:10.638 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.831 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:10.646 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.848 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:10.662 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.849 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:10.664 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:30.857 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:10.672 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.128 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:29.942 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.132 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:29.947 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.154 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:29.969 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.181 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:29.996 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.183 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:29.998 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.183 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:29.947 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.187 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:29.951 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.206 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:30.021 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.207 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:29.971 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.828 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:30.592 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.829 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:30.593 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.836 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:30.601 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.883 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:30.698 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.884 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:30.699 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.892 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:30.707 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.932 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:30.747 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.933 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:30.748 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:23:50.941 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:30.756 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.182 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:49.996 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.186 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:50.001 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.188 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:49.952 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.190 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:49.955 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.208 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:50.022 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.212 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:49.976 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.238 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:50.053 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.242 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:50.057 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.269 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:50.084 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.850 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:50.615 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.851 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:50.616 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.859 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:43:50.623 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.930 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:50.745 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.931 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:50.746 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.940 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:43:50.755 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.968 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:50.783 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.969 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:50.784 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:10.980 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:43:50.795 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:30.254 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:10.068 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:30.256 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:10.071 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:30.275 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:10.090 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:30.303 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:10.118 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:30.306 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:10.121 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:30.307 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:10.071 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:30.310 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:10.074 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:30.327 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:10.142 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:30.329 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:10.093 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:30.970 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:10.734 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:30.971 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:10.735 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:30.979 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:10.744 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:31.061 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:10.875 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:31.061 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:10.876 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:31.072 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:10.887 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:31.088 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:10.903 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:31.089 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:10.904 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:31.097 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:10.912 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:41.634 INFO:tasks.ceph.osd.3:Stopped 2024-02-06T22:24:41.635 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-02-06T22:24:41.635 INFO:teuthology.orchestra.run:waiting for 300 2024-02-06T22:24:41.647 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T22:24:41.638+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 34802) UID: 0 2024-02-06T22:24:41.648 INFO:tasks.ceph.osd.4.smithi135.stderr:2024-02-06T22:24:41.639+0000 b47a640 -1 osd.4 1360 *** Got signal Terminated *** 2024-02-06T22:24:43.635 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:23.450 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:43.644 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:23.459 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:43.657 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:23.421 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:43.659 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:23.424 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:43.666 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:23.481 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:43.715 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:23.530 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:43.727 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:23.542 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:43.742 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:23.557 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:43.773 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:23.537 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:47.738 INFO:tasks.ceph.osd.4:Stopped 2024-02-06T22:24:47.738 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-02-06T22:24:47.738 INFO:teuthology.orchestra.run:waiting for 300 2024-02-06T22:24:47.752 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T22:24:47.742+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 34803) UID: 0 2024-02-06T22:24:47.752 INFO:tasks.ceph.osd.5.smithi135.stderr:2024-02-06T22:24:47.743+0000 b47a640 -1 osd.5 1362 *** Got signal Terminated *** 2024-02-06T22:24:50.340 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:30.155 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:50.342 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:30.158 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:50.360 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:30.175 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:50.391 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:30.155 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:50.392 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:30.157 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:50.407 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:30.171 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:50.409 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:30.224 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:50.415 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:30.230 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:50.435 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:30.249 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:51.010 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:30.774 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:51.011 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:30.775 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:51.021 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:30.785 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:51.130 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:30.945 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:51.131 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:30.946 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:51.140 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:30.955 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:51.156 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:30.971 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:51.158 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:30.973 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:24:51.166 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:30.981 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:10.401 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:50.216 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:10.406 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:50.221 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:10.427 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:50.242 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:10.453 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:50.218 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:10.457 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:50.221 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:10.459 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:50.274 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:10.464 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:50.278 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:10.475 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:50.240 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:10.487 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:50.302 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:11.083 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:50.847 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:11.084 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:50.848 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:11.092 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:44:50.857 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:11.107 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:50.922 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:11.108 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:50.923 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:11.116 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:44:50.931 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:11.154 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:50.969 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:11.155 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:50.970 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:11.164 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:44:50.979 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:30.456 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:10.270 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:30.460 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:10.275 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:30.480 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:10.295 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:30.506 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:10.270 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:30.509 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:10.273 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:30.517 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:10.332 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:30.519 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:10.334 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:30.528 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:10.292 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:30.538 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:10.353 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:31.127 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:10.942 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:31.128 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:10.943 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:31.137 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:10.952 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:31.139 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:10.903 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:31.140 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:10.904 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:31.148 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:10.912 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:31.152 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:10.967 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:31.153 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:10.968 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:31.161 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:10.976 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:50.523 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:30.337 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:50.525 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:30.340 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:50.545 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:30.360 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:50.574 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:30.339 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:50.576 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:30.340 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:50.576 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:30.391 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:50.581 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:30.396 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:50.594 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:30.358 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:50.598 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:30.413 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:51.266 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:31.030 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:51.267 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:31.031 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:51.269 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:31.084 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:51.270 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:31.085 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:51.276 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:31.041 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:51.280 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:31.095 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:51.306 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:31.120 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:51.307 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:31.122 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:25:51.316 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:31.131 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:10.579 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:50.393 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:10.581 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:50.396 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:10.600 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:50.415 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:10.625 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:50.389 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:10.627 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:50.392 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:10.632 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:50.447 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:10.637 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:50.452 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:10.648 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:50.413 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:10.659 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:50.474 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:11.135 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:50.950 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:11.136 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:50.951 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:11.145 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:45:50.960 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:11.164 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:50.929 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:11.166 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:50.930 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:11.175 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:45:50.939 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:11.189 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:51.004 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:11.190 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:51.005 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:11.200 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:45:51.015 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:25.361 INFO:tasks.ceph.osd.5:Stopped 2024-02-06T22:26:25.362 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-02-06T22:26:25.362 INFO:teuthology.orchestra.run:waiting for 300 2024-02-06T22:26:25.375 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T22:26:25.366+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 34807) UID: 0 2024-02-06T22:26:25.378 INFO:tasks.ceph.osd.6.smithi135.stderr:2024-02-06T22:26:25.367+0000 b47a640 -1 osd.6 1364 *** Got signal Terminated *** 2024-02-06T22:26:30.632 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:10.447 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:30.637 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:10.451 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:30.657 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:10.472 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:30.681 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:10.445 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:30.681 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:10.446 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:30.699 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:10.463 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:30.705 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:10.520 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:30.707 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:10.522 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:30.725 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:10.540 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:31.380 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:11.144 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:31.381 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:11.146 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:31.390 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:11.155 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:31.397 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:11.212 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:31.399 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:11.214 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:31.408 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:11.223 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:31.445 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:11.259 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:31.446 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:11.261 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:31.455 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:11.270 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:50.686 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:30.501 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:50.690 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:30.505 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:50.709 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:30.524 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:50.733 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:30.497 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:50.737 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:30.502 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:50.741 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:30.556 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:50.748 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:30.563 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:50.755 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:30.519 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:50.766 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:30.581 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:51.432 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:31.197 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:51.433 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:31.198 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:51.437 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:31.252 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:51.438 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:31.253 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:51.442 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:31.207 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:51.448 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:31.263 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:51.479 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:31.294 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:51.480 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:31.296 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:26:51.493 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:31.308 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:10.747 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:50.562 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:10.750 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:50.565 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:10.769 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:50.584 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:10.799 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:50.563 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:10.800 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:50.615 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:10.802 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:50.566 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:10.805 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:50.620 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:10.819 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:50.583 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:10.826 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:50.641 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:11.488 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:51.252 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:11.489 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:51.253 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:11.498 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:46:51.262 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:11.506 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:51.321 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:11.507 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:51.322 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:11.518 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:46:51.333 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:11.551 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:51.366 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:11.552 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:51.367 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:11.562 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:46:51.377 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:13.363 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-02-06T22:27:14.175 INFO:tasks.ceph.osd.6:Stopped 2024-02-06T22:27:14.176 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-02-06T22:27:14.176 INFO:teuthology.orchestra.run:waiting for 300 2024-02-06T22:27:14.189 INFO:tasks.ceph.osd.7.smithi135.stderr:2024-02-06T22:27:14.179+0000 b47a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 34806) UID: 0 2024-02-06T22:27:14.192 INFO:tasks.ceph.osd.7.smithi135.stderr:2024-02-06T22:27:14.180+0000 b47a640 -1 osd.7 1364 *** Got signal Terminated *** 2024-02-06T22:27:18.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-02-06T22:27:30.812 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:47:10.626 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:30.813 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:47:10.628 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:30.833 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:47:10.648 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:30.863 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:47:10.627 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:30.865 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:47:10.629 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:30.873 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:47:10.688 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:30.877 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:47:10.692 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:30.883 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:47:10.647 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:30.895 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:47:10.710 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:31.530 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:47:11.345 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:31.530 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:47:11.294 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:31.531 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:47:11.346 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:31.531 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:47:11.296 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:31.539 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:47:11.303 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:31.539 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:47:11.354 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:31.568 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:47:11.383 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:31.569 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:47:11.385 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:31.577 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:47:11.392 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:50.860 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:47:30.675 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:50.865 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:47:30.680 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:50.888 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:47:30.699 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:50.912 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:47:30.677 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:50.915 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:47:30.680 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:50.917 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:47:30.732 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:50.926 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:47:30.741 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:50.932 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:47:30.697 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:50.943 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:47:30.758 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:55.837 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:47:35.652 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:55.889 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:47:35.704 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:55.903 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:47:35.718 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:55.934 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:47:35.698 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:55.936 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:47:35.700 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:55.939 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:47:35.754 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:55.945 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:47:35.760 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:55.947 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:47:35.711 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:55.960 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:47:35.775 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:58.735 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:47:38.550 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:58.740 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:47:38.555 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:58.754 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:47:38.569 34518== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:58.794 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:47:38.609 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:58.796 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:47:38.611 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:58.808 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:47:38.572 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:58.809 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:47:38.574 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:58.810 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:47:38.625 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:27:58.823 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:47:38.588 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:28:02.989 INFO:tasks.ceph.osd.7:Stopped 2024-02-06T22:28:02.990 INFO:teuthology.misc:Shutting down mgr daemons... 2024-02-06T22:28:02.990 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-02-06T22:28:02.990 INFO:teuthology.orchestra.run:waiting for 300 2024-02-06T22:28:03.013 INFO:tasks.ceph.mgr.y:Stopped 2024-02-06T22:28:03.014 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-02-06T22:28:03.014 INFO:teuthology.orchestra.run:waiting for 300 2024-02-06T22:28:03.015 INFO:tasks.ceph.mgr.x.smithi135.stderr:2024-02-06T22:28:03.013+0000 7f05e4145640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34264) UID: 0 2024-02-06T22:28:03.043 INFO:tasks.ceph.mgr.x:Stopped 2024-02-06T22:28:03.043 INFO:teuthology.misc:Shutting down mon daemons... 2024-02-06T22:28:03.044 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-02-06T22:28:03.044 INFO:teuthology.orchestra.run:waiting for 300 2024-02-06T22:28:03.068 INFO:tasks.ceph.mon.a.smithi120.stderr:2024-02-06T22:28:03.056+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34485) UID: 0 2024-02-06T22:28:03.070 INFO:tasks.ceph.mon.a.smithi120.stderr:2024-02-06T22:28:03.058+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-06T22:28:57.163 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:48:36.926 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:28:57.163 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:48:36.928 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:28:57.182 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:48:36.946 34266== Warning: unimplemented fcntl command: 1036 2024-02-06T22:28:57.202 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:48:37.017 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:28:57.203 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:48:37.018 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:28:57.221 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:48:37.036 34520== Warning: unimplemented fcntl command: 1036 2024-02-06T22:29:35.141 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:49:14.955 34518== 2024-02-06T22:29:35.141 INFO:tasks.ceph.mon.a.smithi120.stderr:==00:00:49:14.955 34518== Exit program on first error (--exit-on-first-error=yes) 2024-02-06T22:29:35.165 INFO:tasks.ceph.mon.a.smithi120.stderr:daemon-helper: command failed with exit status 42 2024-02-06T22:29:38.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-02-06T22:29:40.671 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-02-06T22:29:40.672 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi120 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-02-06T22:29:40.674 INFO:tasks.ceph.mon.a:Stopped 2024-02-06T22:29:40.674 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-02-06T22:29:40.674 INFO:teuthology.orchestra.run:waiting for 300 2024-02-06T22:29:40.690 INFO:tasks.ceph.mon.c.smithi120.stderr:2024-02-06T22:29:40.677+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34487) UID: 0 2024-02-06T22:29:40.690 INFO:tasks.ceph.mon.c.smithi120.stderr:2024-02-06T22:29:40.679+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-02-06T22:29:41.520 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:49:21.335 34520== 2024-02-06T22:29:41.520 INFO:tasks.ceph.mon.c.smithi120.stderr:==00:00:49:21.335 34520== Exit program on first error (--exit-on-first-error=yes) 2024-02-06T22:29:41.543 INFO:tasks.ceph.mon.c.smithi120.stderr:daemon-helper: command failed with exit status 42 2024-02-06T22:29:43.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-02-06T22:29:43.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-02-06T22:29:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-02-06T22:29:46.776 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi120 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-02-06T22:29:46.776 INFO:tasks.ceph.mon.c:Stopped 2024-02-06T22:29:46.777 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-02-06T22:29:46.777 INFO:teuthology.orchestra.run:waiting for 300 2024-02-06T22:29:46.794 INFO:tasks.ceph.mon.b.smithi135.stderr:2024-02-06T22:29:46.779+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34262) UID: 0 2024-02-06T22:29:46.795 INFO:tasks.ceph.mon.b.smithi135.stderr:2024-02-06T22:29:46.781+0000 9f86640 -1 mon.b@1(leader) e1 *** Got Signal Terminated *** 2024-02-06T22:29:47.437 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:49:27.201 34266== 2024-02-06T22:29:47.437 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:49:27.201 34266== Exit program on first error (--exit-on-first-error=yes) 2024-02-06T22:29:47.458 INFO:tasks.ceph.mon.b.smithi135.stderr:daemon-helper: command failed with exit status 42 2024-02-06T22:29:48.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-02-06T22:29:48.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-02-06T22:29:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-02-06T22:29:52.879 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi135 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-02-06T22:29:52.879 INFO:tasks.ceph.mon.b:Stopped 2024-02-06T22:29:52.879 INFO:tasks.ceph:Checking cluster log for badness... 2024-02-06T22:29:52.880 DEBUG:teuthology.orchestra.run.smithi120:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-02-06T22:29:52.918 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi120.front.sepia.ceph.com 2024-02-06T22:29:52.919 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-02-06T22:29:53.120 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi120.front.sepia.ceph.com 2024-02-06T22:29:53.120 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-02-06T22:29:53.210 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi120.front.sepia.ceph.com 2024-02-06T22:29:53.210 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-02-06T22:29:53.286 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi120.front.sepia.ceph.com 2024-02-06T22:29:53.287 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-02-06T22:29:53.351 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi135.front.sepia.ceph.com 2024-02-06T22:29:53.352 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-02-06T22:29:53.477 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi135.front.sepia.ceph.com 2024-02-06T22:29:53.477 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-02-06T22:29:53.542 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi135.front.sepia.ceph.com 2024-02-06T22:29:53.542 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-02-06T22:29:53.598 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi135.front.sepia.ceph.com 2024-02-06T22:29:53.598 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-02-06T22:29:53.657 INFO:tasks.ceph:Archiving mon data... 2024-02-06T22:29:53.658 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974/data/mon.a.tgz 2024-02-06T22:29:53.659 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2024-02-06T22:29:53.676 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.lgiCjfrcbn 2024-02-06T22:29:53.677 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.lgiCjfrcbn 2024-02-06T22:29:53.838 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.lgiCjfrcbn 2024-02-06T22:29:53.923 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.lgiCjfrcbn is 472KB 2024-02-06T22:29:53.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-02-06T22:29:54.022 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.lgiCjfrcbn 2024-02-06T22:29:54.037 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974/data/mon.c.tgz 2024-02-06T22:29:54.038 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2024-02-06T22:29:54.090 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.BdSrWbMDob 2024-02-06T22:29:54.091 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.BdSrWbMDob 2024-02-06T22:29:54.241 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.BdSrWbMDob 2024-02-06T22:29:54.326 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.BdSrWbMDob is 504KB 2024-02-06T22:29:54.425 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.BdSrWbMDob 2024-02-06T22:29:54.439 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974/data/mon.b.tgz 2024-02-06T22:29:54.440 DEBUG:teuthology.orchestra.run.smithi135:> mktemp 2024-02-06T22:29:54.458 INFO:teuthology.orchestra.run.smithi135.stdout:/tmp/tmp.95yJxe22Km 2024-02-06T22:29:54.458 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.95yJxe22Km 2024-02-06T22:29:54.622 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0666 /tmp/tmp.95yJxe22Km 2024-02-06T22:29:54.708 DEBUG:teuthology.orchestra.remote:smithi135:/tmp/tmp.95yJxe22Km is 504KB 2024-02-06T22:29:54.806 DEBUG:teuthology.orchestra.run.smithi135:> rm -fr /tmp/tmp.95yJxe22Km 2024-02-06T22:29:54.822 INFO:tasks.ceph:Cleaning ceph cluster... 2024-02-06T22:29:54.822 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-02-06T22:29:54.824 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-02-06T22:29:55.187 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-02-06T22:29:55.451 INFO:tasks.ceph:Archiving crash dumps... 2024-02-06T22:29:55.453 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/crash to /home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974/remote/smithi120/crash 2024-02-06T22:29:55.454 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-02-06T22:29:55.484 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/lib/ceph/crash to /home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974/remote/smithi135/crash 2024-02-06T22:29:55.485 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-02-06T22:29:55.511 INFO:tasks.ceph:Compressing logs... 2024-02-06T22:29:55.511 DEBUG:teuthology.orchestra.run.smithi120:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-06T22:29:55.530 DEBUG:teuthology.orchestra.run.smithi135:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-06T22:29:55.557 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39220.log 2024-02-06T22:29:55.559 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40694.log 2024-02-06T22:29:55.559 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41564.log 2024-02-06T22:29:55.559 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108684.log 2024-02-06T22:29:55.559 INFO:teuthology.orchestra.run.smithi120.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39220.log.gz 2024-02-06T22:29:55.559 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72609.log 2024-02-06T22:29:55.559 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108684.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-02-06T22:29:55.559 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.72609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38506.log 2024-02-06T22:29:55.559 INFO:teuthology.orchestra.run.smithi120.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108684.log.gz 2024-02-06T22:29:55.559 INFO:teuthology.orchestra.run.smithi120.stderr:gzip/var/log/ceph/ceph-mon.a.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39213.log 2024-02-06T22:29:55.578 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38652.log 2024-02-06T22:29:55.578 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39213.log: gzip -5 --verbose -- 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38506.log.gz 2024-02-06T22:29:55.579 INFO:teuthology.orchestra.run.smithi120.stderr: /var/log/ceph/ceph-client.admin.39633.log 2024-02-06T22:29:55.579 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38652.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39213.log.gz 2024-02-06T22:29:55.579 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78840.log 2024-02-06T22:29:55.579 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39633.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38652.log.gz 2024-02-06T22:29:55.579 INFO:teuthology.orchestra.run.smithi120.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.108600.log 2024-02-06T22:29:55.579 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.78840.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39633.log.gz 2024-02-06T22:29:55.579 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108180.log 2024-02-06T22:29:55.579 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39462.log 2024-02-06T22:29:55.579 INFO:teuthology.orchestra.run.smithi120.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.72609.log.gz 2024-02-06T22:29:55.579 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108180.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.108600.log.gz 2024-02-06T22:29:55.579 INFO:teuthology.orchestra.run.smithi120.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41033.log 2024-02-06T22:29:55.579 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39462.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108180.log.gz 2024-02-06T22:29:55.579 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40635.log 2024-02-06T22:29:55.579 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41033.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-02-06T22:29:55.580 INFO:teuthology.orchestra.run.smithi120.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39462.log.gz 2024-02-06T22:29:55.580 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5/var/log/ceph/ceph-client.admin.40635.log: --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-02-06T22:29:55.580 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/valgrind/mon.c.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.78840.log.gz 2024-02-06T22:29:55.580 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-02-06T22:29:55.580 INFO:teuthology.orchestra.run.smithi120.stderr: -- /var/log/ceph/valgrind/osd.0.log 2024-02-06T22:29:55.580 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-02-06T22:29:55.580 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/valgrind/osd.2.log: /var/log/ceph/valgrind/osd.0.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-02-06T22:29:55.580 INFO:teuthology.orchestra.run.smithi120.stderr:gzip 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-02-06T22:29:55.580 INFO:teuthology.orchestra.run.smithi120.stderr: -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-02-06T22:29:55.580 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 64.0% 2024-02-06T22:29:55.580 INFO:teuthology.orchestra.run.smithi120.stderr: -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-02-06T22:29:55.580 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108558.log 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-02-06T22:29:55.580 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:29:55.580 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/valgrind/osd.3.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40694.log.gz 2024-02-06T22:29:55.581 INFO:teuthology.orchestra.run.smithi120.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-02-06T22:29:55.581 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41330.log 2024-02-06T22:29:55.581 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68921.log 2024-02-06T22:29:55.581 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39242.log 2024-02-06T22:29:55.581 INFO:teuthology.orchestra.run.smithi120.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108558.log.gz 2024-02-06T22:29:55.581 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.68921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41329.log 2024-02-06T22:29:55.581 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41011.log 2024-02-06T22:29:55.581 INFO:teuthology.orchestra.run.smithi120.stderr: 89.3%/var/log/ceph/ceph-client.admin.39242.log: -- replaced with /var/log/ceph/ceph-client.admin.68921.log.gz 2024-02-06T22:29:55.581 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41329.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39242.log.gz 2024-02-06T22:29:55.581 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107886.log 2024-02-06T22:29:55.581 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39842.log 2024-02-06T22:29:55.581 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.107886.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.41330.log.gz 2024-02-06T22:29:55.581 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41312.log 2024-02-06T22:29:55.581 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39842.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.107886.log.gz 2024-02-06T22:29:55.582 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40004.log 2024-02-06T22:29:55.582 INFO:teuthology.orchestra.run.smithi120.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39842.log.gz 2024-02-06T22:29:55.582 INFO:teuthology.orchestra.run.smithi120.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.41033.log.gz/var/log/ceph/ceph-client.admin.41312.log: 2024-02-06T22:29:55.582 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39211.log 2024-02-06T22:29:55.582 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41363.log 2024-02-06T22:29:55.582 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39211.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40004.log.gz 2024-02-06T22:29:55.582 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38920.log 2024-02-06T22:29:55.582 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41363.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39211.log.gz 2024-02-06T22:29:55.582 INFO:teuthology.orchestra.run.smithi120.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.41329.log.gz 2024-02-06T22:29:55.582 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36746.log 2024-02-06T22:29:55.582 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41360.log 2024-02-06T22:29:55.582 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38920.log: /var/log/ceph/ceph-client.admin.36746.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38920.log.gz 2024-02-06T22:29:55.582 INFO:teuthology.orchestra.run.smithi120.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36746.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41702.log 2024-02-06T22:29:55.582 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:29:55.582 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41360.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.41312.log.gz 2024-02-06T22:29:55.583 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100484.log 2024-02-06T22:29:55.583 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39217.log 2024-02-06T22:29:55.583 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.100484.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.41360.log.gz 2024-02-06T22:29:55.601 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-02-06T22:29:55.601 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34671.log 2024-02-06T22:29:55.601 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-02-06T22:29:55.601 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.34671.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-02-06T22:29:55.601 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-02-06T22:29:55.601 INFO:teuthology.orchestra.run.smithi135.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-02-06T22:29:55.601 INFO:teuthology.orchestra.run.smithi135.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34671.log.gz 2024-02-06T22:29:55.601 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-02-06T22:29:55.601 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/osd.6.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-02-06T22:29:55.601 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-02-06T22:29:55.601 INFO:teuthology.orchestra.run.smithi135.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-02-06T22:29:55.601 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34459.log 2024-02-06T22:29:55.601 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34419.log 2024-02-06T22:29:55.602 INFO:teuthology.orchestra.run.smithi135.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-02-06T22:29:55.602 INFO:teuthology.orchestra.run.smithi135.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-02-06T22:29:55.602 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.34459.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-02-06T22:29:55.602 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.34419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34539.log 2024-02-06T22:29:55.602 INFO:teuthology.orchestra.run.smithi135.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34459.log.gz 2024-02-06T22:29:55.602 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-02-06T22:29:55.602 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.34539.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34419.log.gz 2024-02-06T22:29:55.602 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-02-06T22:29:55.602 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-02-06T22:29:55.602 INFO:teuthology.orchestra.run.smithi135.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34539.log.gz 2024-02-06T22:29:55.602 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34499.log 2024-02-06T22:29:55.602 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34620.log 2024-02-06T22:29:55.602 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.34499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34711.log 2024-02-06T22:29:55.602 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.34620.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-02-06T22:29:55.602 INFO:teuthology.orchestra.run.smithi135.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34499.log.gz 2024-02-06T22:29:55.603 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.34711.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34620.log.gz 2024-02-06T22:29:55.603 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34711.log.gz 2024-02-06T22:29:55.603 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T22:29:55.603 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-02-06T22:29:55.603 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34580.log 2024-02-06T22:29:55.603 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-mon.b.log: 92.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-02-06T22:29:55.603 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.34580.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34580.log.gz 2024-02-06T22:29:55.603 INFO:teuthology.orchestra.run.smithi135.stderr: 87.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-02-06T22:29:55.613 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108306.log 2024-02-06T22:29:55.613 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39217.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39217.log.gz 2024-02-06T22:29:55.613 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100482.log 2024-02-06T22:29:55.613 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108306.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108306.log.gz 2024-02-06T22:29:55.613 INFO:teuthology.orchestra.run.smithi120.stderr: 91.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38701.log 2024-02-06T22:29:55.613 INFO:teuthology.orchestra.run.smithi120.stderr: -- replaced with /var/log/ceph/ceph-client.admin.100484.log.gz 2024-02-06T22:29:55.613 INFO:teuthology.orchestra.run.smithi120.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41363.log.gz 2024-02-06T22:29:55.613 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.100482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41162.log 2024-02-06T22:29:55.613 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40567.log 2024-02-06T22:29:55.614 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41162.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38701.log.gz 2024-02-06T22:29:55.614 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107970.log 2024-02-06T22:29:55.614 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40567.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.41162.log.gz 2024-02-06T22:29:55.614 INFO:teuthology.orchestra.run.smithi120.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.100482.log.gz 2024-02-06T22:29:55.614 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41283.log 2024-02-06T22:29:55.614 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108348.log 2024-02-06T22:29:55.614 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41283.log: /var/log/ceph/ceph-client.admin.107970.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107970.log.gz 2024-02-06T22:29:55.614 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86459.log 2024-02-06T22:29:55.614 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108348.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108348.log.gz 2024-02-06T22:29:55.614 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107844.log 2024-02-06T22:29:55.614 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.86459.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.40567.log.gz 2024-02-06T22:29:55.614 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41097.log 2024-02-06T22:29:55.614 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.107844.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.107844.log.gz 2024-02-06T22:29:55.614 INFO:teuthology.orchestra.run.smithi120.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.41564.log.gz 2024-02-06T22:29:55.615 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-02-06T22:29:55.615 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108390.log 2024-02-06T22:29:55.615 INFO:teuthology.orchestra.run.smithi120.stderr: 91.8%/var/log/ceph/ceph.audit.log: -- replaced with /var/log/ceph/ceph-client.admin.41283.log.gz 2024-02-06T22:29:55.615 INFO:teuthology.orchestra.run.smithi120.stderr:gzip 90.6% -- replaced with /var/log/ceph/ceph-client.admin.86459.log.gz 2024-02-06T22:29:55.615 INFO:teuthology.orchestra.run.smithi120.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40752.log 2024-02-06T22:29:55.615 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108390.log: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.41011.log.gz 2024-02-06T22:29:55.615 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39745.log 2024-02-06T22:29:55.615 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40752.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.108390.log.gz 2024-02-06T22:29:55.615 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108096.log 2024-02-06T22:29:55.615 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39745.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41097.log.gz 2024-02-06T22:29:55.615 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40548.log 2024-02-06T22:29:55.615 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40770.log 2024-02-06T22:29:55.615 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40548.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39745.log.gz 2024-02-06T22:29:55.615 INFO:teuthology.orchestra.run.smithi120.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108096.log.gz 2024-02-06T22:29:55.615 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39960.log 2024-02-06T22:29:55.616 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40770.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.40548.log.gz 2024-02-06T22:29:55.616 INFO:teuthology.orchestra.run.smithi120.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.40752.log.gz 2024-02-06T22:29:55.616 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41577.log 2024-02-06T22:29:55.616 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40537.log 2024-02-06T22:29:55.616 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41577.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39960.log.gz 2024-02-06T22:29:55.616 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38921.log 2024-02-06T22:29:55.616 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40138.log 2024-02-06T22:29:55.616 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38921.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38921.log.gz 2024-02-06T22:29:55.616 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39572.log 2024-02-06T22:29:55.616 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40138.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40138.log.gz 2024-02-06T22:29:55.616 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108012.log 2024-02-06T22:29:55.616 INFO:teuthology.orchestra.run.smithi120.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41577.log.gz 2024-02-06T22:29:55.616 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39572.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39572.log.gz 2024-02-06T22:29:55.617 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39212.log 2024-02-06T22:29:55.617 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67643.log 2024-02-06T22:29:55.617 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39212.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.108012.log.gz 2024-02-06T22:29:55.617 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35828.log 2024-02-06T22:29:55.617 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.67643.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39212.log.gz 2024-02-06T22:29:55.617 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40471.log 2024-02-06T22:29:55.617 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.35828.log: 95.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-02-06T22:29:55.617 INFO:teuthology.orchestra.run.smithi120.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35828.log.gz 2024-02-06T22:29:55.617 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-02-06T22:29:55.617 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78841.log 2024-02-06T22:29:55.617 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-osd.1.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.67643.log.gz 2024-02-06T22:29:55.617 INFO:teuthology.orchestra.run.smithi120.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.40537.log.gz 2024-02-06T22:29:55.624 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108474.log 2024-02-06T22:29:55.624 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.78841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41179.log 2024-02-06T22:29:55.624 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108474.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108474.log.gz 2024-02-06T22:29:55.625 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40025.log 2024-02-06T22:29:55.626 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41179.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.78841.log.gz 2024-02-06T22:29:55.627 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41668.log 2024-02-06T22:29:55.628 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40025.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40025.log.gz 2024-02-06T22:29:55.628 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34829.log 2024-02-06T22:29:55.630 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41668.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.41702.log.gz 2024-02-06T22:29:55.630 INFO:teuthology.orchestra.run.smithi120.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.41668.log.gz 2024-02-06T22:29:55.631 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40096.log 2024-02-06T22:29:55.631 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.34829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39338.log 2024-02-06T22:29:55.633 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40096.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34829.log.gz 2024-02-06T22:29:55.633 INFO:teuthology.orchestra.run.smithi120.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40096.log.gz 2024-02-06T22:29:55.633 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72612.log 2024-02-06T22:29:55.633 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39338.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39338.log.gz 2024-02-06T22:29:55.634 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41499.log 2024-02-06T22:29:55.634 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.72612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41532.log 2024-02-06T22:29:55.635 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41499.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.72612.log.gz 2024-02-06T22:29:55.635 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40702.log 2024-02-06T22:29:55.635 INFO:teuthology.orchestra.run.smithi120.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41499.log.gz 2024-02-06T22:29:55.641 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41532.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.40471.log.gz 2024-02-06T22:29:55.650 INFO:teuthology.orchestra.run.smithi120.stderr:gzip 90.5% -- replaced with /var/log/ceph/ceph-client.admin.41532.log.gz 2024-02-06T22:29:55.650 INFO:teuthology.orchestra.run.smithi120.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40868.log 2024-02-06T22:29:55.650 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41698.log 2024-02-06T22:29:55.650 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40868.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-02-06T22:29:55.654 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41698.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40868.log.gz 2024-02-06T22:29:55.661 INFO:teuthology.orchestra.run.smithi120.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.41698.log.gz 2024-02-06T22:29:55.663 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108264.log 2024-02-06T22:29:55.663 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38465.log 2024-02-06T22:29:55.663 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108264.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108264.log.gz 2024-02-06T22:29:55.664 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72614.log 2024-02-06T22:29:55.665 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38465.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38465.log.gz 2024-02-06T22:29:55.665 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34856.log 2024-02-06T22:29:55.668 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.72614.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.72614.log.gz 2024-02-06T22:29:55.668 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107928.log 2024-02-06T22:29:55.669 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.34856.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34856.log.gz 2024-02-06T22:29:55.669 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40515.log 2024-02-06T22:29:55.670 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.107928.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.107928.log.gz 2024-02-06T22:29:55.671 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39542.log 2024-02-06T22:29:55.679 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40515.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.40515.log.gz 2024-02-06T22:29:55.680 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108516.log 2024-02-06T22:29:55.680 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39542.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39542.log.gz 2024-02-06T22:29:55.684 INFO:teuthology.orchestra.run.smithi120.stderr: 92.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38606.log 2024-02-06T22:29:55.684 INFO:teuthology.orchestra.run.smithi120.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41179.log.gz 2024-02-06T22:29:55.684 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41604.log 2024-02-06T22:29:55.684 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38606.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108516.log.gz 2024-02-06T22:29:55.684 INFO:teuthology.orchestra.run.smithi120.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38606.log.gz 2024-02-06T22:29:55.684 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67641.log 2024-02-06T22:29:55.684 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41645.log 2024-02-06T22:29:55.687 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.67641.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.67641.log.gz 2024-02-06T22:29:55.687 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40180.log 2024-02-06T22:29:55.691 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41645.log: 94.3% -- replaced with /var/log/ceph/ceph-client.admin.40770.log.gz 2024-02-06T22:29:55.699 INFO:teuthology.orchestra.run.smithi120.stderr: 94.6% -- replaced with /var/log/ceph/ceph-client.admin.41604.log.gz 2024-02-06T22:29:55.700 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38926.log 2024-02-06T22:29:55.700 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40988.log 2024-02-06T22:29:55.701 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38926.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40180.log.gz 2024-02-06T22:29:55.701 INFO:teuthology.orchestra.run.smithi120.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38926.log.gz 2024-02-06T22:29:55.714 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108432.log 2024-02-06T22:29:55.714 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41274.log 2024-02-06T22:29:55.716 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108432.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108432.log.gz 2024-02-06T22:29:55.718 INFO:teuthology.orchestra.run.smithi120.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.41645.log.gz 2024-02-06T22:29:55.718 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41536.log 2024-02-06T22:29:55.718 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40993.log 2024-02-06T22:29:55.719 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41274.log: /var/log/ceph/ceph-client.admin.41536.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41274.log.gz 2024-02-06T22:29:55.719 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38085.log 2024-02-06T22:29:55.728 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40993.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.40993.log.gz 2024-02-06T22:29:55.729 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-02-06T22:29:55.729 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38085.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38085.log.gz 2024-02-06T22:29:55.731 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86457.log 2024-02-06T22:29:55.737 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-mgr.y.log: 93.8% -- replaced with /var/log/ceph/ceph-client.admin.40988.log.gz 2024-02-06T22:29:55.737 INFO:teuthology.orchestra.run.smithi120.stderr: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.41536.log.gz 2024-02-06T22:29:55.745 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.86457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39802.log 2024-02-06T22:29:55.745 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34971.log 2024-02-06T22:29:55.746 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39802.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39802.log.gz 2024-02-06T22:29:55.746 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41313.log 2024-02-06T22:29:55.746 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.34971.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.86457.log.gz 2024-02-06T22:29:55.748 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86461.log 2024-02-06T22:29:55.748 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41313.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34971.log.gz 2024-02-06T22:29:55.748 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34527.log 2024-02-06T22:29:55.749 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.86461.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.41313.log.gz 2024-02-06T22:29:55.749 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41183.log 2024-02-06T22:29:55.751 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.34527.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.34527.log.gz 2024-02-06T22:29:55.751 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86460.log 2024-02-06T22:29:55.751 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41183.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.86461.log.gz 2024-02-06T22:29:55.752 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40610.log 2024-02-06T22:29:55.754 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.86460.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.86460.log.gz 2024-02-06T22:29:55.755 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40778.log 2024-02-06T22:29:55.760 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40610.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.41183.log.gz 2024-02-06T22:29:55.760 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38562.log 2024-02-06T22:29:55.761 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40778.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.40610.log.gz 2024-02-06T22:29:55.761 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108726.log 2024-02-06T22:29:55.762 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38562.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38562.log.gz 2024-02-06T22:29:55.762 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86458.log 2024-02-06T22:29:55.763 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108726.log: 94.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-02-06T22:29:55.763 INFO:teuthology.orchestra.run.smithi120.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108726.log.gz 2024-02-06T22:29:55.764 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39671.log 2024-02-06T22:29:55.766 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.86458.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.86458.log.gz 2024-02-06T22:29:55.766 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68920.log 2024-02-06T22:29:55.766 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38744.log 2024-02-06T22:29:55.767 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39671.log: /var/log/ceph/ceph-client.admin.68920.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39671.log.gz 2024-02-06T22:29:55.768 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-02-06T22:29:55.768 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38744.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38744.log.gz 2024-02-06T22:29:55.769 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108138.log 2024-02-06T22:29:55.771 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-osd.0.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.68920.log.gz 2024-02-06T22:29:55.772 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38910.log 2024-02-06T22:29:55.772 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108138.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108138.log.gz 2024-02-06T22:29:55.773 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39184.log 2024-02-06T22:29:55.773 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38910.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38910.log.gz 2024-02-06T22:29:55.775 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39483.log 2024-02-06T22:29:55.775 INFO:teuthology.orchestra.run.smithi120.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.40778.log.gz 2024-02-06T22:29:55.775 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39184.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39184.log.gz 2024-02-06T22:29:55.775 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108766.log 2024-02-06T22:29:55.778 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.39483.log: -- /var/log/ceph/ceph-client.admin.38371.log 2024-02-06T22:29:55.778 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108766.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39483.log.gz 2024-02-06T22:29:55.778 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-02-06T22:29:55.778 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38371.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108766.log.gz 2024-02-06T22:29:55.778 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-02-06T22:29:55.778 INFO:teuthology.orchestra.run.smithi120.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38371.log.gz 2024-02-06T22:29:55.778 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39642.log 2024-02-06T22:29:55.790 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38924.log 2024-02-06T22:29:55.791 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39642.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39642.log.gz 2024-02-06T22:29:55.796 INFO:teuthology.orchestra.run.smithi120.stderr: 93.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-02-06T22:29:55.796 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108222.log 2024-02-06T22:29:55.798 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38924.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38924.log.gz 2024-02-06T22:29:55.798 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41284.log 2024-02-06T22:29:55.799 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108222.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108222.log.gz 2024-02-06T22:29:55.799 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28658.log 2024-02-06T22:29:55.803 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41284.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.41284.log.gz 2024-02-06T22:29:55.803 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39922.log 2024-02-06T22:29:55.803 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph.tmp-client.admin.28658.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28658.log.gz 2024-02-06T22:29:55.803 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38916.log 2024-02-06T22:29:55.804 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39922.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39922.log.gz 2024-02-06T22:29:55.804 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78842.log 2024-02-06T22:29:55.805 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38916.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38916.log.gz 2024-02-06T22:29:55.805 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40560.log 2024-02-06T22:29:55.809 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.78842.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.78842.log.gz 2024-02-06T22:29:55.809 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100485.log 2024-02-06T22:29:55.823 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39862.log 2024-02-06T22:29:55.827 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.100485.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.100485.log.gz 2024-02-06T22:29:55.841 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40490.log 2024-02-06T22:29:55.841 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.39862.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39862.log.gz 2024-02-06T22:29:55.855 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108642.log 2024-02-06T22:29:55.874 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.40490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67642.log 2024-02-06T22:29:55.875 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108642.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.108642.log.gz 2024-02-06T22:29:55.883 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38925.log 2024-02-06T22:29:55.892 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.67642.log: 91.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41643.log 2024-02-06T22:29:55.897 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38925.log: -- replaced with /var/log/ceph/ceph-client.admin.67642.log.gz 2024-02-06T22:29:55.900 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108054.log 2024-02-06T22:29:55.904 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41643.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38925.log.gz 2024-02-06T22:29:55.911 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41039.log 2024-02-06T22:29:55.912 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.108054.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.108054.log.gz 2024-02-06T22:29:55.915 INFO:teuthology.orchestra.run.smithi120.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.40560.log.gz 2024-02-06T22:29:55.929 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38923.log 2024-02-06T22:29:55.938 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.41039.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-02-06T22:29:55.941 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.38923.log: 91.9% 91.0% -- replaced with /var/log/ceph/ceph-client.admin.41643.log.gz 2024-02-06T22:29:55.943 INFO:teuthology.orchestra.run.smithi120.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38923.log.gz 2024-02-06T22:29:55.950 INFO:teuthology.orchestra.run.smithi120.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41039.log.gz 2024-02-06T22:29:55.963 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40842.log 2024-02-06T22:29:55.990 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.40842.log: 92.1% 91.3% -- replaced with /var/log/ceph/ceph-client.admin.40842.log.gz 2024-02-06T22:29:55.995 INFO:teuthology.orchestra.run.smithi120.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40490.log.gz 2024-02-06T22:29:56.123 INFO:teuthology.orchestra.run.smithi135.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-02-06T22:29:56.160 INFO:teuthology.orchestra.run.smithi120.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.40702.log.gz 2024-02-06T22:29:56.928 INFO:teuthology.orchestra.run.smithi120.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40635.log.gz 2024-02-06T22:29:57.497 INFO:teuthology.orchestra.run.smithi120.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-02-06T22:29:58.202 INFO:teuthology.orchestra.run.smithi135.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-02-06T22:29:58.996 INFO:teuthology.orchestra.run.smithi120.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-02-06T22:30:02.651 INFO:teuthology.orchestra.run.smithi135.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-02-06T22:30:02.821 INFO:teuthology.orchestra.run.smithi135.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-02-06T22:30:03.117 INFO:teuthology.orchestra.run.smithi135.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-02-06T22:30:03.288 INFO:teuthology.orchestra.run.smithi120.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-02-06T22:30:03.668 INFO:teuthology.orchestra.run.smithi135.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-02-06T22:30:03.670 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-02-06T22:30:03.670 INFO:teuthology.orchestra.run.smithi135.stderr:real 0m8.103s 2024-02-06T22:30:03.670 INFO:teuthology.orchestra.run.smithi135.stderr:user 0m31.921s 2024-02-06T22:30:03.670 INFO:teuthology.orchestra.run.smithi135.stderr:sys 0m1.162s 2024-02-06T22:30:03.719 INFO:teuthology.orchestra.run.smithi120.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-02-06T22:30:03.780 INFO:teuthology.orchestra.run.smithi120.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-02-06T22:30:04.051 INFO:teuthology.orchestra.run.smithi120.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-02-06T22:30:04.053 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-02-06T22:30:04.053 INFO:teuthology.orchestra.run.smithi120.stderr:real 0m8.510s 2024-02-06T22:30:04.053 INFO:teuthology.orchestra.run.smithi120.stderr:user 0m38.368s 2024-02-06T22:30:04.053 INFO:teuthology.orchestra.run.smithi120.stderr:sys 0m1.384s 2024-02-06T22:30:04.054 INFO:tasks.ceph:Archiving logs... 2024-02-06T22:30:04.054 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/log/ceph to /home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974/remote/smithi120/log 2024-02-06T22:30:04.055 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-06T22:30:09.663 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/log/ceph to /home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974/remote/smithi135/log 2024-02-06T22:30:09.664 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-06T22:30:14.475 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-02-06T22:30:14.476 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-06T22:30:14.486 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_2b35b7e08ec7c184dfd7bd515a9e5acf453f94fc/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-02-06T22:30:14.486 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-06T22:30:14.487 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-06T22:30:14.490 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-06T22:30:14.587 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-06T22:30:14.596 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-06T22:30:14.596 DEBUG:teuthology.orchestra.run.smithi120:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-06T22:30:14.599 DEBUG:teuthology.orchestra.run.smithi135:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-06T22:30:14.613 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-02-06T22:30:14.614 INFO:teuthology.orchestra.run.smithi135.stderr:bash: line 1: ntpq: command not found 2024-02-06T22:30:14.618 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-06T22:30:14.618 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2024-02-06T22:30:14.618 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 +1788us[+1788us] +/- 97ms 2024-02-06T22:30:14.618 INFO:teuthology.orchestra.run.smithi135.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -603us[ -612us] +/- 77ms 2024-02-06T22:30:14.618 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 -866us[ -875us] +/- 78ms 2024-02-06T22:30:14.618 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-06T22:30:14.619 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-06T22:30:14.619 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-02-06T22:30:14.620 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 43 +1820us[+1820us] +/- 97ms 2024-02-06T22:30:14.620 INFO:teuthology.orchestra.run.smithi120.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -576us[ -568us] +/- 77ms 2024-02-06T22:30:14.620 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 -835us[ -827us] +/- 78ms 2024-02-06T22:30:14.620 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-06T22:30:14.620 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-06T22:30:14.629 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-06T22:30:14.629 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-06T22:30:14.638 DEBUG:teuthology.orchestra.run.smithi120:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-06T22:30:14.732 DEBUG:teuthology.orchestra.run.smithi135:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-06T22:30:14.801 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-06T22:30:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-06T22:30:14.832 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-06T22:30:14.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-06T22:30:14.863 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-06T22:30:14.872 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-06T22:30:14.893 INFO:teuthology.task.internal:Duration was 3920.009449 seconds 2024-02-06T22:30:14.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-06T22:30:14.902 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-06T22:30:14.902 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-06T22:30:14.905 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-06T22:30:14.947 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-06T22:30:14.948 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-06T22:30:15.286 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-06T22:30:15.286 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi120.front.sepia.ceph.com 2024-02-06T22:30:15.286 DEBUG:teuthology.orchestra.run.smithi120:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-06T22:30:15.306 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi135.front.sepia.ceph.com 2024-02-06T22:30:15.306 DEBUG:teuthology.orchestra.run.smithi135:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-06T22:30:15.327 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-06T22:30:15.328 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-06T22:30:15.348 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-06T22:30:15.395 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-06T22:30:15.396 DEBUG:teuthology.orchestra.run.smithi120:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-06T22:30:15.416 DEBUG:teuthology.orchestra.run.smithi135:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-06T22:30:15.535 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-06T22:30:15.545 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-06T22:30:15.545 DEBUG:teuthology.orchestra.run.smithi120:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-06T22:30:15.572 DEBUG:teuthology.orchestra.run.smithi135:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-06T22:30:15.598 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-06T22:30:15.608 DEBUG:teuthology.orchestra.run.smithi120:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-06T22:30:15.614 DEBUG:teuthology.orchestra.run.smithi135:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-06T22:30:15.637 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = core 2024-02-06T22:30:15.663 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = core 2024-02-06T22:30:15.688 DEBUG:teuthology.orchestra.run.smithi120:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-06T22:30:15.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-06T22:30:15.708 DEBUG:teuthology.orchestra.run.smithi135:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-06T22:30:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-06T22:30:15.742 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-06T22:30:15.752 INFO:teuthology.task.internal:Transferring archived files... 2024-02-06T22:30:15.753 DEBUG:teuthology.misc:Transferring archived files from smithi120:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974/remote/smithi120 2024-02-06T22:30:15.753 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-06T22:30:15.811 DEBUG:teuthology.misc:Transferring archived files from smithi135:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974/remote/smithi135 2024-02-06T22:30:15.811 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-06T22:30:15.846 INFO:teuthology.task.internal:Removing archive directory... 2024-02-06T22:30:15.846 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-06T22:30:15.849 DEBUG:teuthology.orchestra.run.smithi135:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-06T22:30:15.895 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-06T22:30:15.909 INFO:teuthology.task.internal:Not uploading archives. 2024-02-06T22:30:15.909 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-06T22:30:15.918 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-06T22:30:15.919 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-06T22:30:15.921 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-06T22:30:15.933 INFO:teuthology.orchestra.run.smithi120.stdout: 265534 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 6 22:30 /home/ubuntu/cephtest 2024-02-06T22:30:15.950 INFO:teuthology.orchestra.run.smithi135.stdout: 265293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 6 22:30 /home/ubuntu/cephtest 2024-02-06T22:30:15.951 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-06T22:30:15.961 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-06T22:30:15.972 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-02-06T22:30:15.972 INFO:teuthology.nuke:Checking targets against current locks 2024-02-06T22:30:15.996 DEBUG:teuthology.nuke:shortname: smithi120 2024-02-06T22:30:15.997 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-06T22:30:16.022 DEBUG:teuthology.nuke:shortname: smithi135 2024-02-06T22:30:16.023 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-06T22:30:16.027 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-06 21:16:16.060020', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBERq3tD9axCtMLTUR91xYMSSyspJePB3jBEHX7Vx31cW7gZaKDEvhcIP2WBNlHrWqXXnbvMeAgpeFcKGIFWh2wY='} 2024-02-06T22:30:16.050 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-06_20:55:47-rados-wip-yuri4-testing-2024-02-05-0849-distro-default-smithi/7548974', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-06 21:16:16.062115', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE6IG63DZKKtm3VBcdAQFlU7KcQl66IeJPSg7EJNL9LVQtVfynSBOJoPR4umbHvTxWK70Dd61OqnVyNHcVGH45g='} 2024-02-06T22:30:16.074 INFO:teuthology.orchestra.console.smithi120:Power off 2024-02-06T22:30:16.074 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-06T22:30:16.096 DEBUG:teuthology.orchestra.console.smithi120:power off output: Chassis Power Control: Down/Off 2024-02-06T22:30:16.096 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T22:30:16.116 INFO:teuthology.orchestra.console.smithi135:Power off 2024-02-06T22:30:16.116 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-06T22:30:16.127 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-02-06T22:30:16.145 DEBUG:teuthology.orchestra.console.smithi135:power off output: Chassis Power Control: Down/Off 2024-02-06T22:30:16.145 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T22:30:16.167 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2024-02-06T22:30:20.128 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T22:30:20.168 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T22:30:20.238 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-02-06T22:30:20.277 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2024-02-06T22:30:24.239 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T22:30:24.278 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T22:30:24.353 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-02-06T22:30:24.387 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2024-02-06T22:30:28.357 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T22:30:28.388 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-06T22:30:28.468 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is off 2024-02-06T22:30:28.498 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is off 2024-02-06T22:30:28.569 INFO:teuthology.orchestra.console.smithi120:Power off completed 2024-02-06T22:30:28.599 INFO:teuthology.orchestra.console.smithi135:Power off completed 2024-02-06T22:30:28.703 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 3920.009449005127 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_lflores@teuthology status: fail success: false 2024-02-06T22:30:28.703 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-06T22:30:28.793 INFO:teuthology.run:FAIL